Nov 02 01:31:17.850511 ip-10-0-141-68 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 01:31:18.255657 ip-10-0-141-68 kubenswrapper[2547]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:31:18.255657 ip-10-0-141-68 kubenswrapper[2547]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 01:31:18.255657 ip-10-0-141-68 kubenswrapper[2547]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:31:18.255657 ip-10-0-141-68 kubenswrapper[2547]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:31:18.260833 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260727    2547 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:31:18.260833 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260835    2547 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:31:18.260896 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260841    2547 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:31:18.260896 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260845    2547 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:31:18.260896 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260849    2547 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:31:18.260896 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260853    2547 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:31:18.260896 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260856    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:31:18.260896 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260860    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:31:18.260896 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260863    2547 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:31:18.260896 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260866    2547 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:31:18.260896 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260869    2547 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:31:18.260896 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260871    2547 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:31:18.260896 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260874    2547 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:31:18.260896 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260877    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:31:18.260896 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260879    2547 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:31:18.260896 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260882    2547 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:31:18.260896 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260885    2547 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:31:18.260896 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260887    2547 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:31:18.260896 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260891    2547 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:31:18.260896 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260893    2547 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:31:18.260896 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260896    2547 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:31:18.261384 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260899    2547 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:31:18.261384 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260902    2547 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:31:18.261384 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260905    2547 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:31:18.261384 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260908    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:31:18.261384 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260911    2547 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:31:18.261384 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260914    2547 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:31:18.261384 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260916    2547 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:31:18.261384 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260919    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:31:18.261384 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260921    2547 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:31:18.261384 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260923    2547 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:31:18.261384 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260926    2547 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:31:18.261384 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260928    2547 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:31:18.261384 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260931    2547 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:31:18.261384 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260933    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:31:18.261384 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260936    2547 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:31:18.261384 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260938    2547 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:31:18.261384 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260941    2547 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:31:18.261384 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260943    2547 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:31:18.261384 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260946    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:31:18.261826 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260948    2547 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:31:18.261826 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260950    2547 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:31:18.261826 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260953    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:31:18.261826 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260955    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:31:18.261826 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260958    2547 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:31:18.261826 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260961    2547 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:31:18.261826 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260963    2547 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:31:18.261826 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260965    2547 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:31:18.261826 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260967    2547 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:31:18.261826 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260970    2547 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:31:18.261826 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260972    2547 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:31:18.261826 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260975    2547 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:31:18.261826 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260977    2547 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:31:18.261826 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260980    2547 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:31:18.261826 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260983    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:31:18.261826 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260986    2547 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:31:18.261826 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260988    2547 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:31:18.261826 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260990    2547 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:31:18.261826 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260994    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:31:18.261826 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260996    2547 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:31:18.262314 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.260999    2547 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:31:18.262314 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261002    2547 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:31:18.262314 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261004    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:31:18.262314 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261006    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:31:18.262314 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261009    2547 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:31:18.262314 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261012    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:31:18.262314 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261014    2547 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:31:18.262314 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261017    2547 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:31:18.262314 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261020    2547 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:31:18.262314 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261022    2547 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:31:18.262314 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261025    2547 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:31:18.262314 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261027    2547 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:31:18.262314 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261030    2547 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:31:18.262314 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261032    2547 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:31:18.262314 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261035    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:31:18.262314 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261037    2547 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:31:18.262314 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261040    2547 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:31:18.262314 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261042    2547 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:31:18.262314 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261045    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:31:18.262314 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261047    2547 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:31:18.262771 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261050    2547 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:31:18.262771 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261052    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:31:18.262771 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261054    2547 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:31:18.262771 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261057    2547 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:31:18.262771 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261059    2547 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:31:18.262771 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261063    2547 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:31:18.262771 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261066    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:31:18.262771 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261068    2547 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:31:18.262771 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261071    2547 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:31:18.262771 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261073    2547 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:31:18.262771 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261075    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:31:18.262771 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261079    2547 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:31:18.262771 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261081    2547 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:31:18.262771 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261084    2547 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:31:18.262771 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261086    2547 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:31:18.262771 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261088    2547 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:31:18.262771 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261091    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:31:18.262771 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261094    2547 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:31:18.262771 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261098    2547 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:31:18.262771 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261100    2547 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:31:18.263256 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261103    2547 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:31:18.263256 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261105    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:31:18.263256 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261107    2547 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:31:18.263256 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261110    2547 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:31:18.263256 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261112    2547 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:31:18.263256 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261114    2547 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:31:18.263256 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261923    2547 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:31:18.263256 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261931    2547 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:31:18.263256 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261934    2547 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:31:18.263256 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261937    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:31:18.263256 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261940    2547 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:31:18.263256 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261942    2547 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:31:18.263256 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261945    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:31:18.263256 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261948    2547 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:31:18.263256 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261951    2547 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:31:18.263256 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261954    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:31:18.263256 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261956    2547 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:31:18.263256 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261959    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:31:18.263256 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261962    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:31:18.263256 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261965    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:31:18.263777 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261968    2547 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:31:18.263777 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261971    2547 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:31:18.263777 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261973    2547 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:31:18.263777 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261977    2547 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:31:18.263777 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261981    2547 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:31:18.263777 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261984    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:31:18.263777 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261987    2547 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:31:18.263777 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261989    2547 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:31:18.263777 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261992    2547 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:31:18.263777 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261994    2547 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:31:18.263777 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261997    2547 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:31:18.263777 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.261999    2547 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:31:18.263777 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262002    2547 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:31:18.263777 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262004    2547 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:31:18.263777 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262007    2547 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:31:18.263777 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262009    2547 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:31:18.263777 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262012    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:31:18.263777 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262014    2547 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:31:18.263777 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262017    2547 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:31:18.264251 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262020    2547 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:31:18.264251 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262022    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:31:18.264251 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262025    2547 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:31:18.264251 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262028    2547 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:31:18.264251 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262030    2547 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:31:18.264251 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262033    2547 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:31:18.264251 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262035    2547 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:31:18.264251 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262038    2547 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:31:18.264251 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262041    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:31:18.264251 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262043    2547 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:31:18.264251 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262045    2547 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:31:18.264251 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262049    2547 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:31:18.264251 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262053    2547 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:31:18.264251 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262057    2547 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:31:18.264251 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262060    2547 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:31:18.264251 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262063    2547 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:31:18.264251 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262066    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:31:18.264251 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262068    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:31:18.264735 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262071    2547 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:31:18.264735 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262074    2547 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:31:18.264735 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262077    2547 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:31:18.264735 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262080    2547 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:31:18.264735 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262082    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:31:18.264735 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262085    2547 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:31:18.264735 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262087    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:31:18.264735 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262090    2547 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:31:18.264735 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262093    2547 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:31:18.264735 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262095    2547 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:31:18.264735 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262098    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:31:18.264735 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262100    2547 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:31:18.264735 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262103    2547 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:31:18.264735 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262106    2547 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:31:18.264735 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262108    2547 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:31:18.264735 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262111    2547 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:31:18.264735 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262113    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:31:18.264735 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262116    2547 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:31:18.264735 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262118    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:31:18.264735 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262121    2547 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:31:18.265210 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262124    2547 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:31:18.265210 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262126    2547 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:31:18.265210 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262129    2547 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:31:18.265210 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262131    2547 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:31:18.265210 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262134    2547 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:31:18.265210 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262137    2547 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:31:18.265210 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262140    2547 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:31:18.265210 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262142    2547 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:31:18.265210 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262145    2547 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:31:18.265210 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262147    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:31:18.265210 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262150    2547 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:31:18.265210 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262152    2547 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:31:18.265210 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262155    2547 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:31:18.265210 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262157    2547 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:31:18.265210 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262160    2547 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:31:18.265210 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262163    2547 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:31:18.265210 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262165    2547 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:31:18.265210 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262168    2547 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:31:18.265210 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262170    2547 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:31:18.265210 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262172    2547 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:31:18.265694 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262175    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:31:18.265694 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262177    2547 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:31:18.265694 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262179    2547 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:31:18.265694 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262182    2547 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:31:18.265694 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262184    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:31:18.265694 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262189    2547 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:31:18.265694 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262191    2547 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:31:18.265694 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262194    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:31:18.265694 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262196    2547 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:31:18.265694 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262200    2547 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:31:18.265694 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262202    2547 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:31:18.265694 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262205    2547 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:31:18.265694 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262208    2547 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:31:18.265694 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262210    2547 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:31:18.265694 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262213    2547 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:31:18.265694 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262747    2547 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:31:18.265694 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262754    2547 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:31:18.265694 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262758    2547 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:31:18.265694 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262761    2547 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:31:18.266135 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262764    2547 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:31:18.266135 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262766    2547 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:31:18.266135 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262769    2547 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:31:18.266135 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262772    2547 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:31:18.266135 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262774    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:31:18.266135 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262777    2547 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:31:18.266135 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262787    2547 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:31:18.266135 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262790    2547 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:31:18.266135 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262793    2547 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:31:18.266135 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262796    2547 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:31:18.266135 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262799    2547 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:31:18.266135 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262801    2547 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:31:18.266135 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262804    2547 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:31:18.266135 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262806    2547 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:31:18.266135 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262809    2547 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:31:18.266135 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262811    2547 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:31:18.266135 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262813    2547 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:31:18.266135 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262816    2547 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:31:18.266135 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262821    2547 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:31:18.266135 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262823    2547 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:31:18.266632 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262826    2547 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:31:18.266632 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262828    2547 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:31:18.266632 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262831    2547 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:31:18.266632 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262834    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:31:18.266632 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262836    2547 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:31:18.266632 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262839    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:31:18.266632 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262842    2547 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:31:18.266632 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262844    2547 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:31:18.266632 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262847    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:31:18.266632 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262849    2547 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:31:18.266632 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262852    2547 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:31:18.266632 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262854    2547 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:31:18.266632 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262857    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:31:18.266632 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262860    2547 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:31:18.266632 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262863    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:31:18.266632 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262865    2547 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:31:18.266632 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262867    2547 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:31:18.266632 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262870    2547 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:31:18.266632 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262872    2547 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:31:18.267077 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262880    2547 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:31:18.267077 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262883    2547 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:31:18.267077 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262886    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:31:18.267077 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262888    2547 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:31:18.267077 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262891    2547 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:31:18.267077 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262893    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:31:18.267077 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262896    2547 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:31:18.267077 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262898    2547 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:31:18.267077 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262901    2547 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:31:18.267077 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262903    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:31:18.267077 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262906    2547 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:31:18.267077 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262911    2547 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:31:18.267077 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262914    2547 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:31:18.267077 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262916    2547 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:31:18.267077 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262919    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:31:18.267077 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262921    2547 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:31:18.267077 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262924    2547 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:31:18.267077 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262926    2547 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:31:18.267077 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262929    2547 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:31:18.267077 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262931    2547 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:31:18.267561 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262934    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:31:18.267561 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262936    2547 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:31:18.267561 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262939    2547 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:31:18.267561 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262941    2547 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:31:18.267561 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262944    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:31:18.267561 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262946    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:31:18.267561 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262949    2547 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:31:18.267561 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262951    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:31:18.267561 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262954    2547 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:31:18.267561 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262956    2547 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:31:18.267561 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262959    2547 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:31:18.267561 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262967    2547 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:31:18.267561 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262970    2547 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:31:18.267561 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262977    2547 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:31:18.267561 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262980    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:31:18.267561 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262983    2547 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:31:18.267561 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262985    2547 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:31:18.267561 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262987    2547 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:31:18.267561 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262990    2547 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:31:18.267561 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262992    2547 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:31:18.268028 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262994    2547 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:31:18.268028 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262997    2547 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:31:18.268028 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.262999    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:31:18.268028 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.263003    2547 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:31:18.268028 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.263006    2547 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:31:18.268028 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.263008    2547 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:31:18.268028 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.263011    2547 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:31:18.268028 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.263014    2547 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:31:18.268028 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.263017    2547 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:31:18.268028 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.263020    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:31:18.268028 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.263022    2547 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:31:18.268028 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.263025    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:31:18.268028 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.263027    2547 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:31:18.268028 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.263030    2547 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:31:18.268028 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.263032    2547 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:31:18.268028 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.263035    2547 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:31:18.268028 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.263037    2547 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:31:18.268028 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.263039    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:31:18.268028 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.263042    2547 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:31:18.268480 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.263044    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:31:18.268480 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.263049    2547 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:31:18.268480 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.263053    2547 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:31:18.268480 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.263056    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:31:18.268480 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264000    2547 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 01:31:18.268480 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264010    2547 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 01:31:18.268480 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264016    2547 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 01:31:18.268480 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264023    2547 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 01:31:18.268480 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264027    2547 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 01:31:18.268480 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264030    2547 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 01:31:18.268480 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264035    2547 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 01:31:18.268480 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264039    2547 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 01:31:18.268480 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264042    2547 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 01:31:18.268480 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264046    2547 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 01:31:18.268480 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264049    2547 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 01:31:18.268480 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264053    2547 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 01:31:18.268480 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264056    2547 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 01:31:18.268480 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264060    2547 flags.go:64] FLAG: --cgroup-root=""
Nov 02 01:31:18.268480 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264063    2547 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 01:31:18.268480 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264066    2547 flags.go:64] FLAG: --client-ca-file=""
Nov 02 01:31:18.268480 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264069    2547 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 01:31:18.268480 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264072    2547 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 01:31:18.268480 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264077    2547 flags.go:64] FLAG: --cluster-domain=""
Nov 02 01:31:18.269054 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264080    2547 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 01:31:18.269054 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264083    2547 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 01:31:18.269054 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264086    2547 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 01:31:18.269054 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264090    2547 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 01:31:18.269054 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264094    2547 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 01:31:18.269054 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264097    2547 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 01:31:18.269054 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264100    2547 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 01:31:18.269054 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264103    2547 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 01:31:18.269054 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264106    2547 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 01:31:18.269054 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264109    2547 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 01:31:18.269054 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264112    2547 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 01:31:18.269054 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264116    2547 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 01:31:18.269054 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264119    2547 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 01:31:18.269054 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264123    2547 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 01:31:18.269054 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264126    2547 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 01:31:18.269054 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264129    2547 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 01:31:18.269054 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264132    2547 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 01:31:18.269054 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264139    2547 flags.go:64] FLAG: --enable-server="true"
Nov 02 01:31:18.269054 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264142    2547 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 01:31:18.269054 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264146    2547 flags.go:64] FLAG: --event-burst="100"
Nov 02 01:31:18.269054 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264149    2547 flags.go:64] FLAG: --event-qps="50"
Nov 02 01:31:18.269054 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264152    2547 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 01:31:18.269054 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264155    2547 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 01:31:18.269054 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264158    2547 flags.go:64] FLAG: --eviction-hard=""
Nov 02 01:31:18.269702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264163    2547 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 01:31:18.269702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264166    2547 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 01:31:18.269702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264169    2547 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 01:31:18.269702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264172    2547 flags.go:64] FLAG: --eviction-soft=""
Nov 02 01:31:18.269702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264175    2547 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 01:31:18.269702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264178    2547 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 01:31:18.269702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264181    2547 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 01:31:18.269702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264184    2547 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 01:31:18.269702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264187    2547 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 01:31:18.269702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264190    2547 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 01:31:18.269702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264192    2547 flags.go:64] FLAG: --feature-gates=""
Nov 02 01:31:18.269702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264197    2547 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 01:31:18.269702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264200    2547 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 01:31:18.269702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264203    2547 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 01:31:18.269702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264206    2547 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 01:31:18.269702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264209    2547 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 01:31:18.269702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264212    2547 flags.go:64] FLAG: --help="false"
Nov 02 01:31:18.269702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264215    2547 flags.go:64] FLAG: --hostname-override="ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:31:18.269702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264231    2547 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 01:31:18.269702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264234    2547 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 01:31:18.269702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264238    2547 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 01:31:18.269702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264241    2547 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 01:31:18.269702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264244    2547 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 01:31:18.270256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264247    2547 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 01:31:18.270256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264250    2547 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 01:31:18.270256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264253    2547 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 01:31:18.270256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264261    2547 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 01:31:18.270256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264265    2547 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 01:31:18.270256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264268    2547 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 01:31:18.270256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264271    2547 flags.go:64] FLAG: --kube-reserved=""
Nov 02 01:31:18.270256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264274    2547 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 01:31:18.270256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264277    2547 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 01:31:18.270256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264280    2547 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 01:31:18.270256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264282    2547 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 01:31:18.270256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264285    2547 flags.go:64] FLAG: --lock-file=""
Nov 02 01:31:18.270256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264288    2547 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 01:31:18.270256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264291    2547 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 01:31:18.270256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264294    2547 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 01:31:18.270256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264299    2547 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 01:31:18.270256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264302    2547 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 01:31:18.270256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264305    2547 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 01:31:18.270256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264307    2547 flags.go:64] FLAG: --logging-format="text"
Nov 02 01:31:18.270256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264310    2547 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 01:31:18.270256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264314    2547 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 01:31:18.270256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264317    2547 flags.go:64] FLAG: --manifest-url=""
Nov 02 01:31:18.270256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264320    2547 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 01:31:18.270256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264324    2547 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 01:31:18.270256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264327    2547 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264331    2547 flags.go:64] FLAG: --max-pods="110"
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264334    2547 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264337    2547 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264339    2547 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264342    2547 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264345    2547 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264348    2547 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264351    2547 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264358    2547 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264361    2547 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264363    2547 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264374    2547 flags.go:64] FLAG: --pod-cidr=""
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264377    2547 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264380    2547 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264383    2547 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264386    2547 flags.go:64] FLAG: --port="10250"
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264389    2547 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264392    2547 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0dc07e14c333e45e9"
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264395    2547 flags.go:64] FLAG: --qos-reserved=""
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264398    2547 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264401    2547 flags.go:64] FLAG: --register-node="true"
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264404    2547 flags.go:64] FLAG: --register-with-taints=""
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264408    2547 flags.go:64] FLAG: --registry-burst="10"
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264411    2547 flags.go:64] FLAG: --registry-qps="5"
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264413    2547 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 01:31:18.270876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264416    2547 flags.go:64] FLAG: --reserved-memory=""
Nov 02 01:31:18.271529 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264420    2547 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 01:31:18.271529 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264423    2547 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 01:31:18.271529 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264426    2547 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 01:31:18.271529 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264429    2547 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 01:31:18.271529 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264432    2547 flags.go:64] FLAG: --runonce="false"
Nov 02 01:31:18.271529 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264434    2547 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 01:31:18.271529 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264437    2547 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 01:31:18.271529 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264440    2547 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 01:31:18.271529 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264443    2547 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 01:31:18.271529 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264445    2547 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 01:31:18.271529 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264449    2547 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 01:31:18.271529 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264452    2547 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 01:31:18.271529 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264454    2547 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 01:31:18.271529 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264457    2547 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 01:31:18.271529 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264460    2547 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 01:31:18.271529 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264463    2547 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 01:31:18.271529 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264465    2547 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 01:31:18.271529 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264468    2547 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 01:31:18.271529 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264476    2547 flags.go:64] FLAG: --system-cgroups=""
Nov 02 01:31:18.271529 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264481    2547 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 01:31:18.271529 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264487    2547 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 01:31:18.271529 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264490    2547 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 01:31:18.271529 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264492    2547 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 01:31:18.271529 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264499    2547 flags.go:64] FLAG: --tls-min-version=""
Nov 02 01:31:18.271529 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264502    2547 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 01:31:18.272487 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264505    2547 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 01:31:18.272487 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264508    2547 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 01:31:18.272487 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264511    2547 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 01:31:18.272487 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264514    2547 flags.go:64] FLAG: --v="2"
Nov 02 01:31:18.272487 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264518    2547 flags.go:64] FLAG: --version="false"
Nov 02 01:31:18.272487 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264522    2547 flags.go:64] FLAG: --vmodule=""
Nov 02 01:31:18.272487 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264526    2547 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 01:31:18.272487 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264529    2547 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 01:31:18.272487 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264668    2547 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:31:18.272487 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264672    2547 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:31:18.272487 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264675    2547 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:31:18.272487 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264679    2547 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:31:18.272487 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264682    2547 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:31:18.272487 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264685    2547 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:31:18.272487 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264689    2547 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:31:18.272487 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264692    2547 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:31:18.272487 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264695    2547 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:31:18.272487 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264698    2547 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:31:18.272487 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264701    2547 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:31:18.272487 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264704    2547 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:31:18.272487 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264707    2547 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:31:18.273042 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264709    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:31:18.273042 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264712    2547 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:31:18.273042 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264715    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:31:18.273042 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264717    2547 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:31:18.273042 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264720    2547 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:31:18.273042 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264724    2547 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:31:18.273042 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264726    2547 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:31:18.273042 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264729    2547 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:31:18.273042 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264732    2547 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:31:18.273042 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264734    2547 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:31:18.273042 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264737    2547 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:31:18.273042 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264741    2547 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:31:18.273042 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264744    2547 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:31:18.273042 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264746    2547 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:31:18.273042 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264749    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:31:18.273042 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264751    2547 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:31:18.273042 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264754    2547 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:31:18.273042 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264756    2547 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:31:18.273042 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264759    2547 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:31:18.273042 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264761    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:31:18.273536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264764    2547 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:31:18.273536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264766    2547 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:31:18.273536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264769    2547 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:31:18.273536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264771    2547 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:31:18.273536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264774    2547 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:31:18.273536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264776    2547 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:31:18.273536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264779    2547 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:31:18.273536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264781    2547 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:31:18.273536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264784    2547 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:31:18.273536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264787    2547 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:31:18.273536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264789    2547 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:31:18.273536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264791    2547 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:31:18.273536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264794    2547 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:31:18.273536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264797    2547 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:31:18.273536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264799    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:31:18.273536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264801    2547 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:31:18.273536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264804    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:31:18.273536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264808    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:31:18.273536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264811    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:31:18.273536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264814    2547 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:31:18.274001 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264817    2547 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:31:18.274001 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264819    2547 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:31:18.274001 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264822    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:31:18.274001 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264825    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:31:18.274001 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264828    2547 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:31:18.274001 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264830    2547 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:31:18.274001 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264833    2547 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:31:18.274001 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264835    2547 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:31:18.274001 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264838    2547 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:31:18.274001 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264840    2547 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:31:18.274001 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264843    2547 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:31:18.274001 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264845    2547 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:31:18.274001 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264848    2547 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:31:18.274001 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264850    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:31:18.274001 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264853    2547 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:31:18.274001 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264855    2547 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:31:18.274001 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264858    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:31:18.274001 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264860    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:31:18.274001 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264863    2547 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:31:18.274536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264865    2547 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:31:18.274536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264867    2547 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:31:18.274536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264870    2547 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:31:18.274536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264872    2547 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:31:18.274536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264875    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:31:18.274536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264878    2547 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:31:18.274536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264880    2547 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:31:18.274536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264883    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:31:18.274536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264885    2547 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:31:18.274536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264888    2547 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:31:18.274536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264892    2547 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:31:18.274536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264896    2547 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:31:18.274536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264900    2547 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:31:18.274536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264903    2547 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:31:18.274536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264905    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:31:18.274536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264908    2547 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:31:18.274536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264911    2547 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:31:18.274536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264914    2547 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:31:18.274536 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264916    2547 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:31:18.274988 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264919    2547 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:31:18.274988 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264921    2547 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:31:18.274988 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264924    2547 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:31:18.274988 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264927    2547 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:31:18.274988 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264929    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:31:18.274988 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264932    2547 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:31:18.274988 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264934    2547 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:31:18.274988 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264937    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:31:18.274988 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264939    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:31:18.274988 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264942    2547 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:31:18.274988 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264944    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:31:18.274988 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264947    2547 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:31:18.274988 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264949    2547 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:31:18.274988 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264952    2547 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:31:18.274988 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.264954    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:31:18.274988 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.264981    2547 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:31:18.275380 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265016    2547 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:31:18.275380 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265020    2547 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:31:18.275380 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265024    2547 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:31:18.275380 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265026    2547 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:31:18.275380 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265029    2547 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:31:18.275380 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265032    2547 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:31:18.275380 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265034    2547 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:31:18.275380 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265037    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:31:18.275380 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265042    2547 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:31:18.275380 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265045    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:31:18.275380 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265048    2547 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:31:18.275380 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265050    2547 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:31:18.275380 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265053    2547 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:31:18.275380 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265057    2547 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:31:18.275380 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265059    2547 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:31:18.275380 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265062    2547 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:31:18.275380 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265064    2547 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:31:18.275380 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265067    2547 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:31:18.275380 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265069    2547 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:31:18.275380 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265071    2547 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:31:18.275854 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265074    2547 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:31:18.275854 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265076    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:31:18.275854 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265078    2547 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:31:18.275854 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265081    2547 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:31:18.275854 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265083    2547 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:31:18.275854 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265086    2547 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:31:18.275854 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265088    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:31:18.275854 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265091    2547 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:31:18.275854 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265093    2547 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:31:18.275854 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265096    2547 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:31:18.275854 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265098    2547 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:31:18.275854 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265101    2547 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:31:18.275854 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265103    2547 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:31:18.275854 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265105    2547 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:31:18.275854 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265108    2547 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:31:18.275854 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265110    2547 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:31:18.275854 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265113    2547 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:31:18.275854 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265115    2547 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:31:18.275854 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265118    2547 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:31:18.275854 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265120    2547 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:31:18.276347 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265124    2547 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:31:18.276347 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265127    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:31:18.276347 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265129    2547 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:31:18.276347 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265133    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:31:18.276347 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265135    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:31:18.276347 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265139    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:31:18.276347 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265142    2547 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:31:18.276347 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265144    2547 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:31:18.276347 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265147    2547 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:31:18.276347 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265149    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:31:18.276347 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265151    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:31:18.276347 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265154    2547 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:31:18.276347 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265156    2547 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:31:18.276347 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265159    2547 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:31:18.276347 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265161    2547 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:31:18.276347 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265164    2547 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:31:18.276347 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265166    2547 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:31:18.276347 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265169    2547 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:31:18.276347 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265171    2547 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:31:18.276785 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265174    2547 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:31:18.276785 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265176    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:31:18.276785 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265179    2547 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:31:18.276785 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265181    2547 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:31:18.276785 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265184    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:31:18.276785 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265186    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:31:18.276785 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265189    2547 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:31:18.276785 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265191    2547 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:31:18.276785 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265193    2547 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:31:18.276785 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265196    2547 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:31:18.276785 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265198    2547 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:31:18.276785 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265201    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:31:18.276785 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265203    2547 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:31:18.276785 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265207    2547 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:31:18.276785 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265209    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:31:18.276785 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265212    2547 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:31:18.276785 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265215    2547 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:31:18.276785 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265232    2547 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:31:18.276785 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265237    2547 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:31:18.277234 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265241    2547 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:31:18.277234 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265244    2547 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:31:18.277234 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265247    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:31:18.277234 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265250    2547 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:31:18.277234 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265252    2547 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:31:18.277234 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265255    2547 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:31:18.277234 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265257    2547 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:31:18.277234 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265260    2547 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:31:18.277234 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265263    2547 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:31:18.277234 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265266    2547 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:31:18.277234 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265268    2547 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:31:18.277234 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265271    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:31:18.277234 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265273    2547 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:31:18.277234 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265276    2547 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:31:18.277234 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265278    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:31:18.277234 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265281    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:31:18.277234 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265283    2547 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:31:18.277234 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265286    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:31:18.277234 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265289    2547 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:31:18.277234 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265291    2547 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:31:18.277704 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265294    2547 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:31:18.277704 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265296    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:31:18.277704 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265299    2547 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:31:18.277704 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265302    2547 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:31:18.277704 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265305    2547 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:31:18.277704 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265309    2547 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:31:18.277704 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265314    2547 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:31:18.277704 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.265317    2547 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:31:18.277704 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.273988    2547 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 01:31:18.277704 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.274004    2547 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 01:31:18.277704 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.274820    2547 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 01:31:18.277704 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.274827    2547 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 01:31:18.277704 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274889    2547 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:31:18.277704 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274894    2547 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:31:18.277704 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274898    2547 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:31:18.277704 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274901    2547 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:31:18.277704 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274904    2547 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:31:18.277704 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274906    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:31:18.277704 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274909    2547 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:31:18.278141 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274911    2547 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:31:18.278141 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274914    2547 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:31:18.278141 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274916    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:31:18.278141 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274919    2547 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:31:18.278141 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274921    2547 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:31:18.278141 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274924    2547 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:31:18.278141 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274927    2547 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:31:18.278141 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274929    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:31:18.278141 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274932    2547 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:31:18.278141 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274935    2547 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:31:18.278141 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274937    2547 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:31:18.278141 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274945    2547 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:31:18.278141 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274949    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:31:18.278141 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274951    2547 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:31:18.278141 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274954    2547 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:31:18.278141 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274957    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:31:18.278141 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274959    2547 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:31:18.278141 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274962    2547 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:31:18.278141 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274964    2547 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:31:18.278141 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274967    2547 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:31:18.278715 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274970    2547 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:31:18.278715 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274972    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:31:18.278715 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274975    2547 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:31:18.278715 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274978    2547 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:31:18.278715 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274981    2547 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:31:18.278715 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274984    2547 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:31:18.278715 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274986    2547 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:31:18.278715 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274989    2547 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:31:18.278715 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274992    2547 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:31:18.278715 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274994    2547 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:31:18.278715 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.274997    2547 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:31:18.278715 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275000    2547 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:31:18.278715 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275002    2547 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:31:18.278715 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275004    2547 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:31:18.278715 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275007    2547 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:31:18.278715 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275009    2547 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:31:18.278715 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275012    2547 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:31:18.278715 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275015    2547 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:31:18.278715 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275017    2547 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:31:18.278715 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275020    2547 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:31:18.279176 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275022    2547 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:31:18.279176 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275024    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:31:18.279176 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275027    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:31:18.279176 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275030    2547 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:31:18.279176 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275032    2547 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:31:18.279176 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275035    2547 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:31:18.279176 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275037    2547 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:31:18.279176 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275040    2547 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:31:18.279176 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275043    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:31:18.279176 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275045    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:31:18.279176 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275048    2547 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:31:18.279176 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275050    2547 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:31:18.279176 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275052    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:31:18.279176 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275056    2547 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:31:18.279176 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275058    2547 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:31:18.279176 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275061    2547 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:31:18.279176 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275064    2547 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:31:18.279176 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275066    2547 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:31:18.279176 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275069    2547 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:31:18.279633 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275071    2547 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:31:18.279633 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275074    2547 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:31:18.279633 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275077    2547 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:31:18.279633 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275079    2547 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:31:18.279633 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275082    2547 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:31:18.279633 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275084    2547 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:31:18.279633 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275086    2547 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:31:18.279633 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275089    2547 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:31:18.279633 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275093    2547 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:31:18.279633 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275096    2547 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:31:18.279633 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275099    2547 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:31:18.279633 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275102    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:31:18.279633 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275104    2547 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:31:18.279633 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275113    2547 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:31:18.279633 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275116    2547 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:31:18.279633 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275118    2547 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:31:18.279633 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275121    2547 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:31:18.279633 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275124    2547 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:31:18.279633 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275126    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:31:18.280080 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275129    2547 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:31:18.280080 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275132    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:31:18.280080 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275134    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:31:18.280080 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275137    2547 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:31:18.280080 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275139    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:31:18.280080 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275142    2547 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:31:18.280080 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275144    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:31:18.280080 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275147    2547 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:31:18.280080 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275150    2547 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:31:18.280080 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275152    2547 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:31:18.280080 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275156    2547 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:31:18.280080 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275160    2547 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:31:18.280080 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275164    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:31:18.280080 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275166    2547 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:31:18.280080 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275169    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:31:18.280080 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275171    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:31:18.280080 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275174    2547 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:31:18.280080 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275177    2547 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:31:18.280080 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275179    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:31:18.280538 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275181    2547 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:31:18.280538 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275184    2547 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:31:18.280538 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.275213    2547 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:31:18.280538 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275388    2547 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:31:18.280538 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275392    2547 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:31:18.280538 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275395    2547 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:31:18.280538 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275398    2547 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:31:18.280538 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275401    2547 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:31:18.280538 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275405    2547 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:31:18.280538 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275407    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:31:18.280538 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275419    2547 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:31:18.280538 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275422    2547 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:31:18.280538 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275425    2547 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:31:18.280538 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275428    2547 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:31:18.280538 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275430    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:31:18.280538 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275433    2547 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:31:18.280937 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275435    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:31:18.280937 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275438    2547 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:31:18.280937 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275440    2547 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:31:18.280937 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275443    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:31:18.280937 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275446    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:31:18.280937 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275450    2547 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:31:18.280937 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275453    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:31:18.280937 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275455    2547 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:31:18.280937 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275458    2547 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:31:18.280937 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275461    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:31:18.280937 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275463    2547 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:31:18.280937 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275466    2547 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:31:18.280937 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275468    2547 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:31:18.280937 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275471    2547 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:31:18.280937 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275473    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:31:18.280937 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275476    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:31:18.280937 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275478    2547 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:31:18.280937 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275480    2547 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:31:18.280937 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275483    2547 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:31:18.281391 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275485    2547 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:31:18.281391 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275488    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:31:18.281391 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275490    2547 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:31:18.281391 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275493    2547 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:31:18.281391 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275495    2547 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:31:18.281391 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275498    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:31:18.281391 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275501    2547 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:31:18.281391 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275505    2547 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:31:18.281391 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275509    2547 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:31:18.281391 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275512    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:31:18.281391 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275514    2547 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:31:18.281391 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275517    2547 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:31:18.281391 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275519    2547 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:31:18.281391 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275522    2547 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:31:18.281391 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275524    2547 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:31:18.281391 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275527    2547 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:31:18.281391 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275529    2547 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:31:18.281391 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275531    2547 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:31:18.281391 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275535    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:31:18.281829 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275537    2547 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:31:18.281829 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275540    2547 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:31:18.281829 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275542    2547 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:31:18.281829 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275545    2547 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:31:18.281829 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275547    2547 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:31:18.281829 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275549    2547 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:31:18.281829 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275552    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:31:18.281829 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275554    2547 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:31:18.281829 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275557    2547 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:31:18.281829 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275559    2547 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:31:18.281829 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275562    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:31:18.281829 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275565    2547 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:31:18.281829 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275567    2547 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:31:18.281829 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275569    2547 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:31:18.281829 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275572    2547 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:31:18.281829 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275574    2547 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:31:18.281829 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275576    2547 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:31:18.281829 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275579    2547 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:31:18.281829 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275581    2547 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:31:18.281829 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275584    2547 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:31:18.282376 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275587    2547 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:31:18.282376 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275589    2547 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:31:18.282376 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275592    2547 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:31:18.282376 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275594    2547 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:31:18.282376 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275596    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:31:18.282376 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275599    2547 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:31:18.282376 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275601    2547 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:31:18.282376 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275603    2547 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:31:18.282376 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275606    2547 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:31:18.282376 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275609    2547 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:31:18.282376 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275611    2547 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:31:18.282376 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275614    2547 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:31:18.282376 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275617    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:31:18.282376 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275619    2547 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:31:18.282376 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275621    2547 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:31:18.282376 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275624    2547 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:31:18.282376 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275626    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:31:18.282376 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275629    2547 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:31:18.282376 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275631    2547 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:31:18.282376 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275633    2547 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:31:18.282844 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275636    2547 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:31:18.282844 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275638    2547 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:31:18.282844 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275641    2547 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:31:18.282844 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275643    2547 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:31:18.282844 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275645    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:31:18.282844 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275648    2547 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:31:18.282844 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275651    2547 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:31:18.282844 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275654    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:31:18.282844 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275657    2547 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:31:18.282844 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275660    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:31:18.282844 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275662    2547 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:31:18.282844 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275664    2547 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:31:18.282844 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275667    2547 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:31:18.282844 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275670    2547 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:31:18.282844 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275672    2547 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:31:18.283189 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.275697    2547 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:31:18.283189 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275731    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:31:18.283189 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275735    2547 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:31:18.283189 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275738    2547 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:31:18.283189 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275741    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:31:18.283189 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275744    2547 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:31:18.283189 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275747    2547 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:31:18.283189 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275749    2547 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:31:18.283189 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275752    2547 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:31:18.283189 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275755    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:31:18.283189 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275758    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:31:18.283189 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275762    2547 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:31:18.283189 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275765    2547 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:31:18.283189 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275768    2547 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:31:18.283189 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275770    2547 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:31:18.283189 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275773    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:31:18.283581 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275775    2547 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:31:18.283581 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275777    2547 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:31:18.283581 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275780    2547 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:31:18.283581 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275782    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:31:18.283581 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275785    2547 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:31:18.283581 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275788    2547 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:31:18.283581 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275792    2547 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:31:18.283581 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275795    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:31:18.283581 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275798    2547 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:31:18.283581 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275800    2547 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:31:18.283581 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275803    2547 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:31:18.283581 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275806    2547 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:31:18.283581 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275809    2547 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:31:18.283581 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275812    2547 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:31:18.283581 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275814    2547 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:31:18.283581 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275817    2547 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:31:18.283581 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275820    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:31:18.283581 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275823    2547 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:31:18.283581 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275825    2547 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:31:18.284007 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275828    2547 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:31:18.284007 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275830    2547 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:31:18.284007 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275833    2547 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:31:18.284007 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275835    2547 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:31:18.284007 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275837    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:31:18.284007 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275840    2547 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:31:18.284007 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275843    2547 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:31:18.284007 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275845    2547 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:31:18.284007 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275848    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:31:18.284007 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275850    2547 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:31:18.284007 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275852    2547 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:31:18.284007 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275855    2547 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:31:18.284007 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275857    2547 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:31:18.284007 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275860    2547 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:31:18.284007 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275863    2547 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:31:18.284007 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275865    2547 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:31:18.284007 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275868    2547 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:31:18.284007 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275870    2547 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:31:18.284007 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275873    2547 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:31:18.284007 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275875    2547 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:31:18.284531 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275878    2547 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:31:18.284531 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275880    2547 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:31:18.284531 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275883    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:31:18.284531 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275886    2547 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:31:18.284531 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275888    2547 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:31:18.284531 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275890    2547 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:31:18.284531 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275893    2547 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:31:18.284531 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275896    2547 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:31:18.284531 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275898    2547 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:31:18.284531 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275902    2547 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:31:18.284531 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275905    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:31:18.284531 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275907    2547 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:31:18.284531 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275909    2547 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:31:18.284531 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275912    2547 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:31:18.284531 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275914    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:31:18.284531 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275917    2547 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:31:18.284531 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275919    2547 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:31:18.284531 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275921    2547 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:31:18.284531 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275924    2547 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:31:18.284531 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275927    2547 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:31:18.285016 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275929    2547 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:31:18.285016 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275931    2547 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:31:18.285016 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275934    2547 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:31:18.285016 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275936    2547 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:31:18.285016 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275940    2547 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:31:18.285016 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275943    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:31:18.285016 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275945    2547 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:31:18.285016 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275948    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:31:18.285016 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275950    2547 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:31:18.285016 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275953    2547 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:31:18.285016 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275955    2547 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:31:18.285016 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275957    2547 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:31:18.285016 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275959    2547 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:31:18.285016 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275962    2547 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:31:18.285016 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275964    2547 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:31:18.285016 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275967    2547 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:31:18.285016 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275969    2547 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:31:18.285016 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275971    2547 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:31:18.285016 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275974    2547 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:31:18.285501 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275977    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:31:18.285501 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275979    2547 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:31:18.285501 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275982    2547 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:31:18.285501 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275985    2547 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:31:18.285501 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275987    2547 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:31:18.285501 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275990    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:31:18.285501 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275992    2547 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:31:18.285501 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275994    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:31:18.285501 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275997    2547 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:31:18.285501 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.275999    2547 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:31:18.285501 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.276001    2547 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:31:18.285501 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.276005    2547 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:31:18.285501 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:18.276008    2547 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:31:18.285501 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.276069    2547 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 01:31:18.285501 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.281136    2547 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 01:31:18.285501 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.282162    2547 server.go:1014] "Starting client certificate rotation"
Nov 02 01:31:18.285501 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.282258    2547 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 01:31:18.285501 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.282298    2547 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 01:31:18.285920 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.284274    2547 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 01:31:18.285920 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.285409    2547 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 01:31:18.297878 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.297858    2547 log.go:25] "Validated CRI v1 runtime API"
Nov 02 01:31:18.301642 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.301626    2547 log.go:25] "Validated CRI v1 image API"
Nov 02 01:31:18.303186 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.303163    2547 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 01:31:18.307648 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.307625    2547 fs.go:135] Filesystem UUIDs: map[046c2f35-84f7-4bb3-8e7e-7c290fbde1de:/dev/nvme0n1p3 522aea20-40da-41ac-bac0-7a7a80fd7396:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 02 01:31:18.307698 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.307649    2547 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 01:31:18.313412 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.313300    2547 manager.go:217] Machine: {Timestamp:2026-11-02 01:31:18.311883495 +0000 UTC m=+0.363199243 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109942 MemoryCapacity:32812023808 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24141869081b12c005544779740014 SystemUUID:ec241418-6908-1b12-c005-544779740014 BootID:86282386-ad68-4619-a26c-eea327b1570b Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406011904 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:56:ef:13:53:cf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:56:ef:13:53:cf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:d5:1f:8c:a3:ce Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812023808 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 01:31:18.314035 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.314025    2547 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 01:31:18.314118 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.314107    2547 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 01:31:18.317569 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.317543    2547 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 01:31:18.318494 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.317571    2547 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-68.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 01:31:18.318602 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.318500    2547 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 01:31:18.318602 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.318516    2547 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 01:31:18.318602 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.318570    2547 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 01:31:18.318602 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.318589    2547 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 01:31:18.319856 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.319841    2547 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 01:31:18.324288 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.324276    2547 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 01:31:18.324414 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.324405    2547 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 01:31:18.325263 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.325253    2547 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 01:31:18.325302 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.325266    2547 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 01:31:18.325302 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.325276    2547 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 01:31:18.325302 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.325286    2547 kubelet.go:405] "Adding apiserver pod source"
Nov 02 01:31:18.325302 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.325294    2547 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 01:31:18.325601 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.325590    2547 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 01:31:18.327865 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.327840    2547 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 01:31:18.335110 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.335091    2547 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 01:31:18.336073 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.336061    2547 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 01:31:18.336121 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.336091    2547 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 01:31:18.336296 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.336278    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 01:31:18.336296 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.336295    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 01:31:18.336365 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.336301    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 01:31:18.336365 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.336306    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 01:31:18.336365 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.336312    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 01:31:18.336365 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.336318    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 01:31:18.336990 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.336981    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 01:31:18.336990 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.336990    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 01:31:18.337050 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.336998    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 01:31:18.337050 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.337004    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 01:31:18.337050 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.337012    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 01:31:18.337050 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.337022    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 01:31:18.338655 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.338645    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 01:31:18.338655 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.338655    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 01:31:18.342524 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.342500    2547 server.go:1263] "Started kubelet"
Nov 02 01:31:18.342629 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.342582    2547 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 01:31:18.342679 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.342616    2547 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 01:31:18.342717 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.342700    2547 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 01:31:18.343342 ip-10-0-141-68 systemd[1]: Started Kubernetes Kubelet.
Nov 02 01:31:18.344360 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.344298    2547 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 01:31:18.344687 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.344672    2547 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 01:31:18.347918 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:18.347900    2547 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 01:31:18.352289 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.352274    2547 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 01:31:18.352911 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.352893    2547 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 01:31:18.352999 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:18.352982    2547 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-68.us-west-1.compute.internal\" not found"
Nov 02 01:31:18.353722 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.353699    2547 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 01:31:18.353722 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.353701    2547 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 01:31:18.353722 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.353725    2547 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 01:31:18.353924 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.353864    2547 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 01:31:18.353924 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.353872    2547 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 01:31:18.354206 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.354190    2547 factory.go:55] Registering systemd factory
Nov 02 01:31:18.354300 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.354208    2547 factory.go:223] Registration of the systemd container factory successfully
Nov 02 01:31:18.354446 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.354431    2547 factory.go:153] Registering CRI-O factory
Nov 02 01:31:18.354490 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.354451    2547 factory.go:223] Registration of the crio container factory successfully
Nov 02 01:31:18.354533 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.354522    2547 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 01:31:18.354569 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.354545    2547 factory.go:103] Registering Raw factory
Nov 02 01:31:18.354569 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.354559    2547 manager.go:1199] Started watching for new ooms in manager
Nov 02 01:31:18.355642 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.355630    2547 manager.go:319] Starting recovery of all containers
Nov 02 01:31:18.366395 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.366377    2547 manager.go:324] Recovery completed
Nov 02 01:31:18.368032 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:18.368002    2547 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Nov 02 01:31:18.370814 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.370802    2547 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:31:18.372371 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.372349    2547 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:31:18.372431 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.372384    2547 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:31:18.372431 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.372395    2547 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:31:18.372879 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.372867    2547 cpu_manager.go:226] "Starting" policy="none"
Nov 02 01:31:18.372879 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.372877    2547 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:31:18.372964 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.372877    2547 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 01:31:18.372964 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.372959    2547 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 01:31:18.373547 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.373532    2547 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:31:18.373629 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.373561    2547 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:31:18.373629 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.373574    2547 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:31:18.375144 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.375128    2547 policy_none.go:50] "Start"
Nov 02 01:31:18.375213 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.375147    2547 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 01:31:18.375213 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.375160    2547 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 01:31:18.377314 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.377300    2547 policy_none.go:44] "Start"
Nov 02 01:31:18.408859 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.408733    2547 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 01:31:18.414539 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.414527    2547 manager.go:341] "Starting Device Plugin manager"
Nov 02 01:31:18.414613 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:18.414554    2547 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 01:31:18.414613 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.414563    2547 server.go:87] "Starting device plugin registration server"
Nov 02 01:31:18.414801 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.414788    2547 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 01:31:18.414845 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.414806    2547 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 01:31:18.414896 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.414877    2547 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 01:31:18.414969 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.414956    2547 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 01:31:18.415009 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.414970    2547 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 01:31:18.415488 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:18.415463    2547 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 01:31:18.415571 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:18.415520    2547 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-68.us-west-1.compute.internal\" not found"
Nov 02 01:31:18.484609 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.484585    2547 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-68.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 01:31:18.486530 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:18.484584    2547 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-68.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 01:31:18.486530 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:18.484585    2547 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-68.us-west-1.compute.internal.18e4164e93c136a6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-68.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-68.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 01:31:18.342473382 +0000 UTC m=+0.393789130,LastTimestamp:2026-11-02 01:31:18.342473382 +0000 UTC m=+0.393789130,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-68.us-west-1.compute.internal,}"
Nov 02 01:31:18.493520 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.493495    2547 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 01:31:18.494749 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.494733    2547 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 01:31:18.494749 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.494750    2547 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 01:31:18.494866 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.494766    2547 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 01:31:18.494866 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:18.494799    2547 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 02 01:31:18.515118 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.515072    2547 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:31:18.515860 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.515846    2547 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:31:18.515950 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.515874    2547 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:31:18.515950 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.515884    2547 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:31:18.515950 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.515906    2547 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:31:18.558151 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.558121    2547 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 01:31:18.558306 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.558182    2547 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b7xnv"
Nov 02 01:31:18.595374 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.595348    2547 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal"]
Nov 02 01:31:18.595546 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.595429    2547 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:31:18.596439 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.596408    2547 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:31:18.596536 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.596445    2547 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:31:18.596536 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.596455    2547 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:31:18.598660 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.598647    2547 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:31:18.598802 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.598787    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:31:18.598849 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.598819    2547 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:31:18.599400 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.599385    2547 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:31:18.599400 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.599393    2547 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:31:18.599565 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.599419    2547 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:31:18.599565 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.599425    2547 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:31:18.599565 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.599429    2547 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:31:18.599565 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.599438    2547 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:31:18.601475 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.601458    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:31:18.601584 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.601481    2547 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:31:18.602235 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.602207    2547 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:31:18.602312 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.602248    2547 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:31:18.602312 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.602259    2547 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:31:18.635177 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:18.635152    2547 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-68.us-west-1.compute.internal\" not found" node="ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:31:18.637315 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:18.637296    2547 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-68.us-west-1.compute.internal\" not found" node="ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:31:18.643335 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.643318    2547 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:31:18.644642 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.644629    2547 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:31:18.655822 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.655805    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efa1d9dffd42c4f4f5c559910849b4c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal\" (UID: \"efa1d9dffd42c4f4f5c559910849b4c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:31:18.655876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.655835    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5845b2d4146c451ff8e611722d7c3e34-config\") pod \"kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal\" (UID: \"5845b2d4146c451ff8e611722d7c3e34\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:31:18.655876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.655852    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/efa1d9dffd42c4f4f5c559910849b4c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal\" (UID: \"efa1d9dffd42c4f4f5c559910849b4c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:31:18.742836 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.742807    2547 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:31:18.743687 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.743669    2547 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:31:18.743779 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.743765    2547 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:31:18.743840 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:18.743785    2547 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-68.us-west-1.compute.internal\": node \"ip-10-0-141-68.us-west-1.compute.internal\" not found"
Nov 02 01:31:18.757075 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.757056    2547 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:31:18.757126 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.757085    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/efa1d9dffd42c4f4f5c559910849b4c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal\" (UID: \"efa1d9dffd42c4f4f5c559910849b4c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:31:18.757126 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.757108    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efa1d9dffd42c4f4f5c559910849b4c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal\" (UID: \"efa1d9dffd42c4f4f5c559910849b4c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:31:18.757187 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.757129    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5845b2d4146c451ff8e611722d7c3e34-config\") pod \"kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal\" (UID: \"5845b2d4146c451ff8e611722d7c3e34\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:31:18.757187 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.757179    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5845b2d4146c451ff8e611722d7c3e34-config\") pod \"kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal\" (UID: \"5845b2d4146c451ff8e611722d7c3e34\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:31:18.757331 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.757183    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efa1d9dffd42c4f4f5c559910849b4c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal\" (UID: \"efa1d9dffd42c4f4f5c559910849b4c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:31:18.757331 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.757187    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/efa1d9dffd42c4f4f5c559910849b4c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal\" (UID: \"efa1d9dffd42c4f4f5c559910849b4c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:31:18.819085 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.819030    2547 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:31:18.820406 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.820393    2547 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:31:18.877257 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.877236    2547 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:31:18.938260 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.938241    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:31:18.940408 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:18.940394    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:31:19.281490 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.281410    2547 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 01:31:19.281972 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.281523    2547 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 01:31:19.281972 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.281571    2547 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 01:31:19.281972 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.281571    2547 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 01:31:19.281972 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:19.281581    2547 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a86811a309f80445ca4534cfcca7e946-0363337bf0817064.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-141-68.us-west-1.compute.internal.18e4164e9589d596\": read tcp 10.0.141.68:55716->3.142.175.121:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-141-68.us-west-1.compute.internal.18e4164e9589d596  default   5213 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-68.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-68.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-68.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 01:31:18 +0000 UTC,LastTimestamp:2026-11-02 01:31:18.515887892 +0000 UTC m=+0.567203641,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-68.us-west-1.compute.internal,}"
Nov 02 01:31:19.326303 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.326280    2547 apiserver.go:52] "Watching apiserver"
Nov 02 01:31:19.353123 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.353104    2547 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 01:31:19.563131 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.563051    2547 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 01:26:18 +0000 UTC" deadline="2028-08-18 07:04:05.01394214 +0000 UTC"
Nov 02 01:31:19.563131 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.563087    2547 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15725h32m45.450858034s"
Nov 02 01:31:19.591944 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.591921    2547 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 01:31:19.592242 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.592209    2547 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-jtl6l","openshift-network-operator/iptables-alerter-f297t","openshift-multus/network-metrics-daemon-gws4h","openshift-ovn-kubernetes/ovnkube-node-gfkkb","kube-system/kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-vh2dn","openshift-dns/node-resolver-bd696","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c","openshift-image-registry/node-ca-7xs66","openshift-multus/multus-7tt8p","openshift-network-diagnostics/network-check-target-gzq27","kube-system/global-pull-secret-syncer-8vhjc","kube-system/konnectivity-agent-ft5bg"]
Nov 02 01:31:19.594684 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.594670    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:31:19.596666 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.596642    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f297t"
Nov 02 01:31:19.598626 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.598609    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:19.598689 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:19.598669    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gws4h" podUID="841f3428-10c6-4079-a256-cf6e5cd21806"
Nov 02 01:31:19.600713 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.600698    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.602769 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.602755    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.604920 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.604899    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bd696"
Nov 02 01:31:19.608287 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.608022    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:31:19.610375 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.610359    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7xs66"
Nov 02 01:31:19.612517 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.612503    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.614619 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.614601    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:19.614702 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:19.614682    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzq27" podUID="6a2f0fa5-cbce-4695-aa38-e8bbeb720280"
Nov 02 01:31:19.616792 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.616777    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:19.616870 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:19.616835    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vhjc" podUID="b88b4dfb-1523-4701-9388-781cedab823b"
Nov 02 01:31:19.619687 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.619669    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ft5bg"
Nov 02 01:31:19.646894 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.646872    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 01:31:19.646992 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.646913    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 01:31:19.646992 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.646946    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 01:31:19.646992 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.646958    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 01:31:19.647105 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.647004    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wfswr\""
Nov 02 01:31:19.647105 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.647006    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 01:31:19.648147 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.648131    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 01:31:19.648288 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.648273    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:31:19.648348 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.648310    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-87db6\""
Nov 02 01:31:19.648378 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.648344    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 01:31:19.652297 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.652278    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 01:31:19.652582 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.652569    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 01:31:19.652582 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.652573    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 01:31:19.652683 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.652621    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 01:31:19.652683 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.652653    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 01:31:19.652683 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.652677    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 01:31:19.654437 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.654425    2547 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 01:31:19.660436 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.660418    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-run-netns\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.660512 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.660441    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-log-socket\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.660512 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.660458    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-run-ovn-kubernetes\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.660512 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.660472    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-lib-modules\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.660512 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.660486    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-sys-fs\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:31:19.660634 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.660538    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/046dd898-c779-4ce2-84ec-dc91afb8323f-serviceca\") pod \"node-ca-7xs66\" (UID: \"046dd898-c779-4ce2-84ec-dc91afb8323f\") " pod="openshift-image-registry/node-ca-7xs66"
Nov 02 01:31:19.660634 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.660572    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-var-lib-cni-multus\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.660634 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.660599    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nd6c\" (UniqueName: \"kubernetes.io/projected/841f3428-10c6-4079-a256-cf6e5cd21806-kube-api-access-2nd6c\") pod \"network-metrics-daemon-gws4h\" (UID: \"841f3428-10c6-4079-a256-cf6e5cd21806\") " pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:19.660634 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.660615    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-run-systemd\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.660801 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.660647    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:31:19.660801 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.660683    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-etc-selinux\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:31:19.660801 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.660714    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-var-lib-kubelet\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.660801 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.660744    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-slash\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.660801 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.660771    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-cnibin\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:31:19.660969 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.660808    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-device-dir\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:31:19.660969 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.660840    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-run-k8s-cni-cncf-io\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.660969 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.660865    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b88b4dfb-1523-4701-9388-781cedab823b-kubelet-config\") pod \"global-pull-secret-syncer-8vhjc\" (UID: \"b88b4dfb-1523-4701-9388-781cedab823b\") " pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:19.660969 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.660882    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-run-openvswitch\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.660969 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.660901    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-kubernetes\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.660969 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.660926    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-sysctl-d\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.660969 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.660943    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-socket-dir\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:31:19.660969 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.660956    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a39f3ef9-2d60-4da6-b69a-9bdfac039833-env-overrides\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.660969 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.660970    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/65f63e43-859a-4d3a-8393-99724782596b-konnectivity-ca\") pod \"konnectivity-agent-ft5bg\" (UID: \"65f63e43-859a-4d3a-8393-99724782596b\") " pod="kube-system/konnectivity-agent-ft5bg"
Nov 02 01:31:19.661341 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.660990    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:31:19.661341 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661007    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/adb8a888-520e-4ef5-819a-ee9331b0ccd9-iptables-alerter-script\") pod \"iptables-alerter-f297t\" (UID: \"adb8a888-520e-4ef5-819a-ee9331b0ccd9\") " pod="openshift-network-operator/iptables-alerter-f297t"
Nov 02 01:31:19.661341 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661033    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-sysconfig\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.661341 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661051    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-sysctl-conf\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.661341 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661083    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnbp7\" (UniqueName: \"kubernetes.io/projected/046dd898-c779-4ce2-84ec-dc91afb8323f-kube-api-access-nnbp7\") pod \"node-ca-7xs66\" (UID: \"046dd898-c779-4ce2-84ec-dc91afb8323f\") " pod="openshift-image-registry/node-ca-7xs66"
Nov 02 01:31:19.661341 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661114    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-sys\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.661341 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661137    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-var-lib-kubelet\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.661341 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661160    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:31:19.661341 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661186    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-registration-dir\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:31:19.661341 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661209    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-systemd-units\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.661341 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661253    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9f798cfa-1160-4da5-859c-47d81c07f969-hosts-file\") pod \"node-resolver-bd696\" (UID: \"9f798cfa-1160-4da5-859c-47d81c07f969\") " pod="openshift-dns/node-resolver-bd696"
Nov 02 01:31:19.661341 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661271    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/65f63e43-859a-4d3a-8393-99724782596b-agent-certs\") pod \"konnectivity-agent-ft5bg\" (UID: \"65f63e43-859a-4d3a-8393-99724782596b\") " pod="kube-system/konnectivity-agent-ft5bg"
Nov 02 01:31:19.661341 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661285    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vmxb\" (UniqueName: \"kubernetes.io/projected/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-kube-api-access-5vmxb\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:31:19.661341 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661298    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-multus-cni-dir\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.661341 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661314    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-multus-conf-dir\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.661341 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661328    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-etc-openvswitch\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.661896 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661354    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-cni-netd\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.661896 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661370    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st4j9\" (UniqueName: \"kubernetes.io/projected/a39f3ef9-2d60-4da6-b69a-9bdfac039833-kube-api-access-st4j9\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.661896 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661383    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f798cfa-1160-4da5-859c-47d81c07f969-tmp-dir\") pod \"node-resolver-bd696\" (UID: \"9f798cfa-1160-4da5-859c-47d81c07f969\") " pod="openshift-dns/node-resolver-bd696"
Nov 02 01:31:19.661896 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661397    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-cnibin\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.661896 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661410    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-multus-socket-dir-parent\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.661896 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661425    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a39f3ef9-2d60-4da6-b69a-9bdfac039833-ovnkube-config\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.661896 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661448    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a39f3ef9-2d60-4da6-b69a-9bdfac039833-ovnkube-script-lib\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.661896 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661470    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-systemd\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.661896 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661485    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-run-netns\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.661896 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661498    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb7sn\" (UniqueName: \"kubernetes.io/projected/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-kube-api-access-zb7sn\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.661896 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661512    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b88b4dfb-1523-4701-9388-781cedab823b-dbus\") pod \"global-pull-secret-syncer-8vhjc\" (UID: \"b88b4dfb-1523-4701-9388-781cedab823b\") " pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:19.661896 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661526    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a39f3ef9-2d60-4da6-b69a-9bdfac039833-ovn-node-metrics-cert\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.661896 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661543    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9sj2\" (UniqueName: \"kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2\") pod \"network-check-target-gzq27\" (UID: \"6a2f0fa5-cbce-4695-aa38-e8bbeb720280\") " pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:19.661896 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661560    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-tuned\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.661896 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661573    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-cni-bin\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.661896 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661587    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-cni-binary-copy\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.661896 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661599    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-kubelet\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.662440 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661621    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-var-lib-openvswitch\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.662440 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661642    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-run-ovn\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.662440 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661661    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/adb8a888-520e-4ef5-819a-ee9331b0ccd9-host-slash\") pod \"iptables-alerter-f297t\" (UID: \"adb8a888-520e-4ef5-819a-ee9331b0ccd9\") " pod="openshift-network-operator/iptables-alerter-f297t"
Nov 02 01:31:19.662440 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661679    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-modprobe-d\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.662440 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661704    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-hostroot\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.662440 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661722    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret\") pod \"global-pull-secret-syncer-8vhjc\" (UID: \"b88b4dfb-1523-4701-9388-781cedab823b\") " pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:19.662440 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661738    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs\") pod \"network-metrics-daemon-gws4h\" (UID: \"841f3428-10c6-4079-a256-cf6e5cd21806\") " pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:19.662440 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661752    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-node-log\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.662440 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661806    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66fbw\" (UniqueName: \"kubernetes.io/projected/adb8a888-520e-4ef5-819a-ee9331b0ccd9-kube-api-access-66fbw\") pod \"iptables-alerter-f297t\" (UID: \"adb8a888-520e-4ef5-819a-ee9331b0ccd9\") " pod="openshift-network-operator/iptables-alerter-f297t"
Nov 02 01:31:19.662440 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661844    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-run\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.662440 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661880    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-var-lib-cni-bin\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.662440 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661902    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-run-multus-certs\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.662440 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661922    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-host\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.662440 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661946    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-system-cni-dir\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.662440 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661968    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9cm9\" (UniqueName: \"kubernetes.io/projected/9f798cfa-1160-4da5-859c-47d81c07f969-kube-api-access-f9cm9\") pod \"node-resolver-bd696\" (UID: \"9f798cfa-1160-4da5-859c-47d81c07f969\") " pod="openshift-dns/node-resolver-bd696"
Nov 02 01:31:19.662440 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.661991    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbwwx\" (UniqueName: \"kubernetes.io/projected/902d51a8-2407-4f4b-a5d9-134a40ad956c-kube-api-access-kbwwx\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.662853 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.662028    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.662853 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.662053    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-system-cni-dir\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:31:19.662853 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.662068    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:31:19.662853 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.662100    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-multus-daemon-config\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.662853 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.662122    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-etc-kubernetes\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.662853 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.662137    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-cni-binary-copy\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:31:19.662853 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.662154    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f4gw\" (UniqueName: \"kubernetes.io/projected/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-kube-api-access-4f4gw\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:31:19.662853 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.662184    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/902d51a8-2407-4f4b-a5d9-134a40ad956c-tmp\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.662853 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.662199    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/046dd898-c779-4ce2-84ec-dc91afb8323f-host\") pod \"node-ca-7xs66\" (UID: \"046dd898-c779-4ce2-84ec-dc91afb8323f\") " pod="openshift-image-registry/node-ca-7xs66"
Nov 02 01:31:19.693399 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.693383    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xbzw4\""
Nov 02 01:31:19.696472 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.696300    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 01:31:19.696472 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.696306    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:31:19.696472 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.696349    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 01:31:19.696472 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.696361    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 01:31:19.696472 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.696400    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 01:31:19.696472 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.696411    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xv9qq\""
Nov 02 01:31:19.696472 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.696415    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 01:31:19.696472 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.696450    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 01:31:19.696472 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.696455    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z6qvz\""
Nov 02 01:31:19.696472 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.696463    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 01:31:19.697511 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.697497    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 01:31:19.697511 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.697504    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 01:31:19.697984 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.697971    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j52qq\""
Nov 02 01:31:19.698045 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.698018    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vt6x2\""
Nov 02 01:31:19.698079 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.698050    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n6fsn\""
Nov 02 01:31:19.701909 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.701893    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 01:31:19.702024 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.701998    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kk5lr\""
Nov 02 01:31:19.702024 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.702013    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 01:31:19.702112 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.702078    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 01:31:19.744047 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.744025    2547 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 01:31:19.744121 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.744074    2547 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lxngg"
Nov 02 01:31:19.762626 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.762605    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbwwx\" (UniqueName: \"kubernetes.io/projected/902d51a8-2407-4f4b-a5d9-134a40ad956c-kube-api-access-kbwwx\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.762720 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.762633    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.762720 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.762651    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-system-cni-dir\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:31:19.762720 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.762667    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:31:19.762720 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.762683    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-multus-daemon-config\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.762929 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.762722    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.762929 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.762736    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-system-cni-dir\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:31:19.762929 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.762781    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-etc-kubernetes\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.762929 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.762809    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-cni-binary-copy\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:31:19.762929 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.762835    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f4gw\" (UniqueName: \"kubernetes.io/projected/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-kube-api-access-4f4gw\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:31:19.762929 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.762847    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-etc-kubernetes\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.763208 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.762939    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/902d51a8-2407-4f4b-a5d9-134a40ad956c-tmp\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.763208 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.762970    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/046dd898-c779-4ce2-84ec-dc91afb8323f-host\") pod \"node-ca-7xs66\" (UID: \"046dd898-c779-4ce2-84ec-dc91afb8323f\") " pod="openshift-image-registry/node-ca-7xs66"
Nov 02 01:31:19.763208 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.762996    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-run-netns\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.763208 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763020    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-log-socket\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.763208 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763044    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-run-ovn-kubernetes\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.763208 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763067    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/046dd898-c779-4ce2-84ec-dc91afb8323f-host\") pod \"node-ca-7xs66\" (UID: \"046dd898-c779-4ce2-84ec-dc91afb8323f\") " pod="openshift-image-registry/node-ca-7xs66"
Nov 02 01:31:19.763208 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763086    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-run-netns\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.763208 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763100    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-run-ovn-kubernetes\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.763208 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763120    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-lib-modules\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.763208 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763130    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-log-socket\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.763208 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763145    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-sys-fs\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:31:19.763208 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763170    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/046dd898-c779-4ce2-84ec-dc91afb8323f-serviceca\") pod \"node-ca-7xs66\" (UID: \"046dd898-c779-4ce2-84ec-dc91afb8323f\") " pod="openshift-image-registry/node-ca-7xs66"
Nov 02 01:31:19.763208 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763181    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:31:19.763208 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763196    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-var-lib-cni-multus\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.763762 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763242    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nd6c\" (UniqueName: \"kubernetes.io/projected/841f3428-10c6-4079-a256-cf6e5cd21806-kube-api-access-2nd6c\") pod \"network-metrics-daemon-gws4h\" (UID: \"841f3428-10c6-4079-a256-cf6e5cd21806\") " pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:19.763762 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763253    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-sys-fs\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:31:19.763762 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763259    2547 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 01:31:19.763762 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763276    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-lib-modules\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.763762 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763295    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-var-lib-cni-multus\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.763762 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763300    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-run-systemd\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.763762 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763267    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-run-systemd\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.763762 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763331    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-multus-daemon-config\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.763762 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763334    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:31:19.763762 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763360    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-cni-binary-copy\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:31:19.763762 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763391    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-etc-selinux\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:31:19.763762 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763428    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-var-lib-kubelet\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.763762 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763452    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-slash\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.763762 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763474    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-cnibin\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:31:19.763762 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763492    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-var-lib-kubelet\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.763762 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763511    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-slash\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.763762 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763530    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-etc-selinux\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:31:19.764645 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763532    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-device-dir\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:31:19.764645 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763572    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-device-dir\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:31:19.764645 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763572    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-run-k8s-cni-cncf-io\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.764645 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763538    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-cnibin\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:31:19.764645 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763601    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-run-k8s-cni-cncf-io\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.764645 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763605    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b88b4dfb-1523-4701-9388-781cedab823b-kubelet-config\") pod \"global-pull-secret-syncer-8vhjc\" (UID: \"b88b4dfb-1523-4701-9388-781cedab823b\") " pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:19.764645 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763630    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-run-openvswitch\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.764645 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763632    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/046dd898-c779-4ce2-84ec-dc91afb8323f-serviceca\") pod \"node-ca-7xs66\" (UID: \"046dd898-c779-4ce2-84ec-dc91afb8323f\") " pod="openshift-image-registry/node-ca-7xs66"
Nov 02 01:31:19.764645 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763645    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-kubernetes\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.764645 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763659    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-sysctl-d\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.764645 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763675    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-socket-dir\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:31:19.764645 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763692    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-run-openvswitch\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.764645 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763709    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-kubernetes\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.764645 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763729    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a39f3ef9-2d60-4da6-b69a-9bdfac039833-env-overrides\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.764645 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763733    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:31:19.764645 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763746    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/65f63e43-859a-4d3a-8393-99724782596b-konnectivity-ca\") pod \"konnectivity-agent-ft5bg\" (UID: \"65f63e43-859a-4d3a-8393-99724782596b\") " pod="kube-system/konnectivity-agent-ft5bg"
Nov 02 01:31:19.764645 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763745    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b88b4dfb-1523-4701-9388-781cedab823b-kubelet-config\") pod \"global-pull-secret-syncer-8vhjc\" (UID: \"b88b4dfb-1523-4701-9388-781cedab823b\") " pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:19.765426 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763773    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-sysctl-d\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.765426 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763798    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-socket-dir\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:31:19.765426 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763808    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:31:19.765426 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763832    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/adb8a888-520e-4ef5-819a-ee9331b0ccd9-iptables-alerter-script\") pod \"iptables-alerter-f297t\" (UID: \"adb8a888-520e-4ef5-819a-ee9331b0ccd9\") " pod="openshift-network-operator/iptables-alerter-f297t"
Nov 02 01:31:19.765426 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763854    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-sysconfig\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.765426 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763875    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-sysctl-conf\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.765426 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763898    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnbp7\" (UniqueName: \"kubernetes.io/projected/046dd898-c779-4ce2-84ec-dc91afb8323f-kube-api-access-nnbp7\") pod \"node-ca-7xs66\" (UID: \"046dd898-c779-4ce2-84ec-dc91afb8323f\") " pod="openshift-image-registry/node-ca-7xs66"
Nov 02 01:31:19.765426 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763921    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-sys\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.765426 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763931    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-sysconfig\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.765426 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763943    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-var-lib-kubelet\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.765426 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763932    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:31:19.765426 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763967    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:31:19.765426 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763990    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-sys\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.765426 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.763994    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-registration-dir\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:31:19.765426 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764017    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-systemd-units\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.765426 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764024    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-sysctl-conf\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.765426 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764039    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9f798cfa-1160-4da5-859c-47d81c07f969-hosts-file\") pod \"node-resolver-bd696\" (UID: \"9f798cfa-1160-4da5-859c-47d81c07f969\") " pod="openshift-dns/node-resolver-bd696"
Nov 02 01:31:19.766153 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764061    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/65f63e43-859a-4d3a-8393-99724782596b-agent-certs\") pod \"konnectivity-agent-ft5bg\" (UID: \"65f63e43-859a-4d3a-8393-99724782596b\") " pod="kube-system/konnectivity-agent-ft5bg"
Nov 02 01:31:19.766153 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764077    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-registration-dir\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:31:19.766153 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764076    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-systemd-units\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.766153 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764086    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmxb\" (UniqueName: \"kubernetes.io/projected/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-kube-api-access-5vmxb\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:31:19.766153 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764120    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-multus-cni-dir\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.766153 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764125    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:31:19.766153 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764041    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-var-lib-kubelet\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.766153 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764149    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a39f3ef9-2d60-4da6-b69a-9bdfac039833-env-overrides\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.766153 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764184    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9f798cfa-1160-4da5-859c-47d81c07f969-hosts-file\") pod \"node-resolver-bd696\" (UID: \"9f798cfa-1160-4da5-859c-47d81c07f969\") " pod="openshift-dns/node-resolver-bd696"
Nov 02 01:31:19.766153 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764209    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-multus-conf-dir\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.766153 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764247    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-etc-openvswitch\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.766153 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764276    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-cni-netd\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.766153 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764281    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/65f63e43-859a-4d3a-8393-99724782596b-konnectivity-ca\") pod \"konnectivity-agent-ft5bg\" (UID: \"65f63e43-859a-4d3a-8393-99724782596b\") " pod="kube-system/konnectivity-agent-ft5bg"
Nov 02 01:31:19.766153 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764296    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-multus-cni-dir\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.766153 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764300    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st4j9\" (UniqueName: \"kubernetes.io/projected/a39f3ef9-2d60-4da6-b69a-9bdfac039833-kube-api-access-st4j9\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.766153 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764311    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-multus-conf-dir\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.766153 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764329    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f798cfa-1160-4da5-859c-47d81c07f969-tmp-dir\") pod \"node-resolver-bd696\" (UID: \"9f798cfa-1160-4da5-859c-47d81c07f969\") " pod="openshift-dns/node-resolver-bd696"
Nov 02 01:31:19.766153 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764337    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-cni-netd\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.766957 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764329    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-etc-openvswitch\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.766957 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764352    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-cnibin\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.766957 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764379    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-multus-socket-dir-parent\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.766957 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764391    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-cnibin\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.766957 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764403    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a39f3ef9-2d60-4da6-b69a-9bdfac039833-ovnkube-config\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.766957 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764427    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a39f3ef9-2d60-4da6-b69a-9bdfac039833-ovnkube-script-lib\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.766957 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764441    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-multus-socket-dir-parent\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.766957 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764450    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-systemd\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.766957 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764474    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-run-netns\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.766957 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764496    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb7sn\" (UniqueName: \"kubernetes.io/projected/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-kube-api-access-zb7sn\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.766957 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764520    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b88b4dfb-1523-4701-9388-781cedab823b-dbus\") pod \"global-pull-secret-syncer-8vhjc\" (UID: \"b88b4dfb-1523-4701-9388-781cedab823b\") " pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:19.766957 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764543    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a39f3ef9-2d60-4da6-b69a-9bdfac039833-ovn-node-metrics-cert\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.766957 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764568    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9sj2\" (UniqueName: \"kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2\") pod \"network-check-target-gzq27\" (UID: \"6a2f0fa5-cbce-4695-aa38-e8bbeb720280\") " pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:19.766957 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764572    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f798cfa-1160-4da5-859c-47d81c07f969-tmp-dir\") pod \"node-resolver-bd696\" (UID: \"9f798cfa-1160-4da5-859c-47d81c07f969\") " pod="openshift-dns/node-resolver-bd696"
Nov 02 01:31:19.766957 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764592    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-tuned\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.766957 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764630    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/adb8a888-520e-4ef5-819a-ee9331b0ccd9-iptables-alerter-script\") pod \"iptables-alerter-f297t\" (UID: \"adb8a888-520e-4ef5-819a-ee9331b0ccd9\") " pod="openshift-network-operator/iptables-alerter-f297t"
Nov 02 01:31:19.766957 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764784    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-run-netns\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.766957 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764846    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-systemd\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.767675 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764895    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-cni-bin\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.767675 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764953    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b88b4dfb-1523-4701-9388-781cedab823b-dbus\") pod \"global-pull-secret-syncer-8vhjc\" (UID: \"b88b4dfb-1523-4701-9388-781cedab823b\") " pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:19.767675 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.764985    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-cni-bin\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.767675 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765018    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-cni-binary-copy\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.767675 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765042    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-kubelet\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.767675 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765064    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-var-lib-openvswitch\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.767675 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765092    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-run-ovn\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.767675 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765118    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/adb8a888-520e-4ef5-819a-ee9331b0ccd9-host-slash\") pod \"iptables-alerter-f297t\" (UID: \"adb8a888-520e-4ef5-819a-ee9331b0ccd9\") " pod="openshift-network-operator/iptables-alerter-f297t"
Nov 02 01:31:19.767675 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765124    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-kubelet\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.767675 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765144    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-modprobe-d\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.767675 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765171    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-hostroot\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.767675 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765184    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-var-lib-openvswitch\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.767675 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765194    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret\") pod \"global-pull-secret-syncer-8vhjc\" (UID: \"b88b4dfb-1523-4701-9388-781cedab823b\") " pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:19.767675 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765237    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs\") pod \"network-metrics-daemon-gws4h\" (UID: \"841f3428-10c6-4079-a256-cf6e5cd21806\") " pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:19.767675 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765264    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-node-log\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.767675 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765285    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-modprobe-d\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.767675 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765289    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66fbw\" (UniqueName: \"kubernetes.io/projected/adb8a888-520e-4ef5-819a-ee9331b0ccd9-kube-api-access-66fbw\") pod \"iptables-alerter-f297t\" (UID: \"adb8a888-520e-4ef5-819a-ee9331b0ccd9\") " pod="openshift-network-operator/iptables-alerter-f297t"
Nov 02 01:31:19.767675 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765313    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-run\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.768238 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765328    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/adb8a888-520e-4ef5-819a-ee9331b0ccd9-host-slash\") pod \"iptables-alerter-f297t\" (UID: \"adb8a888-520e-4ef5-819a-ee9331b0ccd9\") " pod="openshift-network-operator/iptables-alerter-f297t"
Nov 02 01:31:19.768238 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765348    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-var-lib-cni-bin\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.768238 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765374    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-run-multus-certs\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.768238 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765398    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-host\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.768238 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765421    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-system-cni-dir\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.768238 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765443    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9cm9\" (UniqueName: \"kubernetes.io/projected/9f798cfa-1160-4da5-859c-47d81c07f969-kube-api-access-f9cm9\") pod \"node-resolver-bd696\" (UID: \"9f798cfa-1160-4da5-859c-47d81c07f969\") " pod="openshift-dns/node-resolver-bd696"
Nov 02 01:31:19.768238 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:19.765585    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:19.768238 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:19.765600    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:19.768238 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:19.765655    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret podName:b88b4dfb-1523-4701-9388-781cedab823b nodeName:}" failed. No retries permitted until 2026-11-02 01:31:20.265635663 +0000 UTC m=+2.316951417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret") pod "global-pull-secret-syncer-8vhjc" (UID: "b88b4dfb-1523-4701-9388-781cedab823b") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:19.768238 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765660    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-node-log\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.768238 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:19.765674    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs podName:841f3428-10c6-4079-a256-cf6e5cd21806 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:20.26566432 +0000 UTC m=+2.316980062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs") pod "network-metrics-daemon-gws4h" (UID: "841f3428-10c6-4079-a256-cf6e5cd21806") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:19.768238 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765682    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-run-ovn\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.768238 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765710    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-run-multus-certs\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.768238 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765734    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-var-lib-cni-bin\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.768238 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765748    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-hostroot\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.768238 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765738    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-run\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.768238 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765741    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-host\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.768704 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765811    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-system-cni-dir\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.768704 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.765894    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a39f3ef9-2d60-4da6-b69a-9bdfac039833-ovnkube-config\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.768704 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.766060    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a39f3ef9-2d60-4da6-b69a-9bdfac039833-ovnkube-script-lib\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.768704 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.766063    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-cni-binary-copy\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.768704 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.766821    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/902d51a8-2407-4f4b-a5d9-134a40ad956c-tmp\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.768704 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.766924    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/65f63e43-859a-4d3a-8393-99724782596b-agent-certs\") pod \"konnectivity-agent-ft5bg\" (UID: \"65f63e43-859a-4d3a-8393-99724782596b\") " pod="kube-system/konnectivity-agent-ft5bg"
Nov 02 01:31:19.768704 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.767191    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-tuned\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.768704 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.768156    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a39f3ef9-2d60-4da6-b69a-9bdfac039833-ovn-node-metrics-cert\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.826960 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.826903    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnbp7\" (UniqueName: \"kubernetes.io/projected/046dd898-c779-4ce2-84ec-dc91afb8323f-kube-api-access-nnbp7\") pod \"node-ca-7xs66\" (UID: \"046dd898-c779-4ce2-84ec-dc91afb8323f\") " pod="openshift-image-registry/node-ca-7xs66"
Nov 02 01:31:19.827095 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.827011    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vmxb\" (UniqueName: \"kubernetes.io/projected/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-kube-api-access-5vmxb\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:31:19.827186 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.827167    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f4gw\" (UniqueName: \"kubernetes.io/projected/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-kube-api-access-4f4gw\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:31:19.827348 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.827332    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nd6c\" (UniqueName: \"kubernetes.io/projected/841f3428-10c6-4079-a256-cf6e5cd21806-kube-api-access-2nd6c\") pod \"network-metrics-daemon-gws4h\" (UID: \"841f3428-10c6-4079-a256-cf6e5cd21806\") " pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:19.827587 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.827573    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st4j9\" (UniqueName: \"kubernetes.io/projected/a39f3ef9-2d60-4da6-b69a-9bdfac039833-kube-api-access-st4j9\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.827650 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.827631    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbwwx\" (UniqueName: \"kubernetes.io/projected/902d51a8-2407-4f4b-a5d9-134a40ad956c-kube-api-access-kbwwx\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.829329 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:19.829306    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:31:19.829449 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:19.829336    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:31:19.829449 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:19.829349    2547 projected.go:196] Error preparing data for projected volume kube-api-access-c9sj2 for pod openshift-network-diagnostics/network-check-target-gzq27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:19.829449 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:19.829408    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2 podName:6a2f0fa5-cbce-4695-aa38-e8bbeb720280 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:20.329389208 +0000 UTC m=+2.380704958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-c9sj2" (UniqueName: "kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2") pod "network-check-target-gzq27" (UID: "6a2f0fa5-cbce-4695-aa38-e8bbeb720280") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:19.831304 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.831285    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66fbw\" (UniqueName: \"kubernetes.io/projected/adb8a888-520e-4ef5-819a-ee9331b0ccd9-kube-api-access-66fbw\") pod \"iptables-alerter-f297t\" (UID: \"adb8a888-520e-4ef5-819a-ee9331b0ccd9\") " pod="openshift-network-operator/iptables-alerter-f297t"
Nov 02 01:31:19.831638 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.831619    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9cm9\" (UniqueName: \"kubernetes.io/projected/9f798cfa-1160-4da5-859c-47d81c07f969-kube-api-access-f9cm9\") pod \"node-resolver-bd696\" (UID: \"9f798cfa-1160-4da5-859c-47d81c07f969\") " pod="openshift-dns/node-resolver-bd696"
Nov 02 01:31:19.831638 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.831632    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb7sn\" (UniqueName: \"kubernetes.io/projected/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-kube-api-access-zb7sn\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:19.932052 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.932024    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:31:19.939173 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.939149    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f297t"
Nov 02 01:31:19.955880 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.955861    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:31:19.960439 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.960423    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:19.974895 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.974877    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bd696"
Nov 02 01:31:19.995644 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:19.995623    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:31:20.004256 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:20.004208    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7xs66"
Nov 02 01:31:20.006868 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:20.006834    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f798cfa_1160_4da5_859c_47d81c07f969.slice/crio-9662347313a235d1d0e72eb393df4285943e8fbfe4cee5d50247970837213a04 WatchSource:0}: Error finding container 9662347313a235d1d0e72eb393df4285943e8fbfe4cee5d50247970837213a04: Status 404 returned error can't find the container with id 9662347313a235d1d0e72eb393df4285943e8fbfe4cee5d50247970837213a04
Nov 02 01:31:20.009751 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:20.009731    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa1d9dffd42c4f4f5c559910849b4c9.slice/crio-94511ea34e79b2676f7c1c9811db65435d12c7d56f939ce4661f88d3f3ff18ef WatchSource:0}: Error finding container 94511ea34e79b2676f7c1c9811db65435d12c7d56f939ce4661f88d3f3ff18ef: Status 404 returned error can't find the container with id 94511ea34e79b2676f7c1c9811db65435d12c7d56f939ce4661f88d3f3ff18ef
Nov 02 01:31:20.011841 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:20.011825    2547 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:31:20.019620 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:20.019598    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d19e74b_e519_4c0a_a3aa_956c6b3bcb28.slice/crio-173c5a4c499cae3df6ed358eb93376aa21a1fa387ba8ee04b6aad1f8dab9f9cb WatchSource:0}: Error finding container 173c5a4c499cae3df6ed358eb93376aa21a1fa387ba8ee04b6aad1f8dab9f9cb: Status 404 returned error can't find the container with id 173c5a4c499cae3df6ed358eb93376aa21a1fa387ba8ee04b6aad1f8dab9f9cb
Nov 02 01:31:20.020014 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:20.019997    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046dd898_c779_4ce2_84ec_dc91afb8323f.slice/crio-56b9dbaa2578f01ef84a655f37e6ee1c3a1c3602959ecb6b552615b7300c9990 WatchSource:0}: Error finding container 56b9dbaa2578f01ef84a655f37e6ee1c3a1c3602959ecb6b552615b7300c9990: Status 404 returned error can't find the container with id 56b9dbaa2578f01ef84a655f37e6ee1c3a1c3602959ecb6b552615b7300c9990
Nov 02 01:31:20.022787 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:20.022763    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5845b2d4146c451ff8e611722d7c3e34.slice/crio-374a8710e7ed9580be3a4b357b8d10b27f6d9121cb57ac4c27d1e0737cdcf3fc WatchSource:0}: Error finding container 374a8710e7ed9580be3a4b357b8d10b27f6d9121cb57ac4c27d1e0737cdcf3fc: Status 404 returned error can't find the container with id 374a8710e7ed9580be3a4b357b8d10b27f6d9121cb57ac4c27d1e0737cdcf3fc
Nov 02 01:31:20.023593 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:20.023569    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7tt8p"
Nov 02 01:31:20.028510 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:20.028496    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ft5bg"
Nov 02 01:31:20.032199 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:20.032179    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a38f9a_7d31_4a28_8a1b_8653ac3b7a20.slice/crio-e98988375935e3f025b17c237709d84c2173ddc0314bbecd7df3a7c758006e8d WatchSource:0}: Error finding container e98988375935e3f025b17c237709d84c2173ddc0314bbecd7df3a7c758006e8d: Status 404 returned error can't find the container with id e98988375935e3f025b17c237709d84c2173ddc0314bbecd7df3a7c758006e8d
Nov 02 01:31:20.036618 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:20.036598    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f63e43_859a_4d3a_8393_99724782596b.slice/crio-1a39fcc2e1764800b057926da3ca002baee535c03f38487c841ddb9afed6adeb WatchSource:0}: Error finding container 1a39fcc2e1764800b057926da3ca002baee535c03f38487c841ddb9afed6adeb: Status 404 returned error can't find the container with id 1a39fcc2e1764800b057926da3ca002baee535c03f38487c841ddb9afed6adeb
Nov 02 01:31:20.174413 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:20.174327    2547 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:31:20.268652 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:20.268619    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret\") pod \"global-pull-secret-syncer-8vhjc\" (UID: \"b88b4dfb-1523-4701-9388-781cedab823b\") " pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:20.268652 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:20.268649    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs\") pod \"network-metrics-daemon-gws4h\" (UID: \"841f3428-10c6-4079-a256-cf6e5cd21806\") " pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:20.268851 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:20.268739    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:20.268851 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:20.268751    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:20.268851 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:20.268786    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs podName:841f3428-10c6-4079-a256-cf6e5cd21806 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:21.268774448 +0000 UTC m=+3.320090184 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs") pod "network-metrics-daemon-gws4h" (UID: "841f3428-10c6-4079-a256-cf6e5cd21806") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:20.268851 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:20.268803    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret podName:b88b4dfb-1523-4701-9388-781cedab823b nodeName:}" failed. No retries permitted until 2026-11-02 01:31:21.26879159 +0000 UTC m=+3.320107330 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret") pod "global-pull-secret-syncer-8vhjc" (UID: "b88b4dfb-1523-4701-9388-781cedab823b") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:20.369378 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:20.369342    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9sj2\" (UniqueName: \"kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2\") pod \"network-check-target-gzq27\" (UID: \"6a2f0fa5-cbce-4695-aa38-e8bbeb720280\") " pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:20.369989 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:20.369489    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:31:20.369989 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:20.369503    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:31:20.369989 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:20.369512    2547 projected.go:196] Error preparing data for projected volume kube-api-access-c9sj2 for pod openshift-network-diagnostics/network-check-target-gzq27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:20.369989 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:20.369562    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2 podName:6a2f0fa5-cbce-4695-aa38-e8bbeb720280 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:21.369549885 +0000 UTC m=+3.420865635 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-c9sj2" (UniqueName: "kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2") pod "network-check-target-gzq27" (UID: "6a2f0fa5-cbce-4695-aa38-e8bbeb720280") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:20.499138 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:20.499034    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ft5bg" event={"ID":"65f63e43-859a-4d3a-8393-99724782596b","Type":"ContainerStarted","Data":"1a39fcc2e1764800b057926da3ca002baee535c03f38487c841ddb9afed6adeb"}
Nov 02 01:31:20.500162 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:20.500137    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7tt8p" event={"ID":"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20","Type":"ContainerStarted","Data":"e98988375935e3f025b17c237709d84c2173ddc0314bbecd7df3a7c758006e8d"}
Nov 02 01:31:20.501219 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:20.501197    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal" event={"ID":"5845b2d4146c451ff8e611722d7c3e34","Type":"ContainerStarted","Data":"374a8710e7ed9580be3a4b357b8d10b27f6d9121cb57ac4c27d1e0737cdcf3fc"}
Nov 02 01:31:20.502499 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:20.502376    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7xs66" event={"ID":"046dd898-c779-4ce2-84ec-dc91afb8323f","Type":"ContainerStarted","Data":"56b9dbaa2578f01ef84a655f37e6ee1c3a1c3602959ecb6b552615b7300c9990"}
Nov 02 01:31:20.504606 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:20.504574    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c" event={"ID":"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28","Type":"ContainerStarted","Data":"173c5a4c499cae3df6ed358eb93376aa21a1fa387ba8ee04b6aad1f8dab9f9cb"}
Nov 02 01:31:20.506567 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:20.506543    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bd696" event={"ID":"9f798cfa-1160-4da5-859c-47d81c07f969","Type":"ContainerStarted","Data":"9662347313a235d1d0e72eb393df4285943e8fbfe4cee5d50247970837213a04"}
Nov 02 01:31:20.507613 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:20.507593    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal" event={"ID":"efa1d9dffd42c4f4f5c559910849b4c9","Type":"ContainerStarted","Data":"94511ea34e79b2676f7c1c9811db65435d12c7d56f939ce4661f88d3f3ff18ef"}
Nov 02 01:31:20.571955 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:20.571931    2547 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:31:20.664907 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:20.664879    2547 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:31:20.744521 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:20.744483    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e02ef8d_7b7b_4de9_a2db_ae32d200a497.slice/crio-8181c311e1c1935dc496d928f2500bd28e19de558f63de49bf74791c75f5bb03 WatchSource:0}: Error finding container 8181c311e1c1935dc496d928f2500bd28e19de558f63de49bf74791c75f5bb03: Status 404 returned error can't find the container with id 8181c311e1c1935dc496d928f2500bd28e19de558f63de49bf74791c75f5bb03
Nov 02 01:31:20.746704 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:20.746647    2547 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 01:26:19 +0000 UTC" deadline="2028-07-04 09:58:47.608654349 +0000 UTC"
Nov 02 01:31:20.746704 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:20.746704    2547 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14648h27m26.861954716s"
Nov 02 01:31:20.767513 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:20.767476    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902d51a8_2407_4f4b_a5d9_134a40ad956c.slice/crio-0d06127b2465ebbbb352b6533e80b8f303a389e2b3d2af40b57fc18c3cd0c9ca WatchSource:0}: Error finding container 0d06127b2465ebbbb352b6533e80b8f303a389e2b3d2af40b57fc18c3cd0c9ca: Status 404 returned error can't find the container with id 0d06127b2465ebbbb352b6533e80b8f303a389e2b3d2af40b57fc18c3cd0c9ca
Nov 02 01:31:20.849190 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:20.849159    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39f3ef9_2d60_4da6_b69a_9bdfac039833.slice/crio-69ae9b34a115e17992488ab6bb6a978d66e0bdaf4fdb424715e2eee95957142d WatchSource:0}: Error finding container 69ae9b34a115e17992488ab6bb6a978d66e0bdaf4fdb424715e2eee95957142d: Status 404 returned error can't find the container with id 69ae9b34a115e17992488ab6bb6a978d66e0bdaf4fdb424715e2eee95957142d
Nov 02 01:31:20.890320 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:31:20.890289    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb8a888_520e_4ef5_819a_ee9331b0ccd9.slice/crio-74a59b76abefddb3b42ae880ccfb8df7538d03ad03fee714c400b67133f82159 WatchSource:0}: Error finding container 74a59b76abefddb3b42ae880ccfb8df7538d03ad03fee714c400b67133f82159: Status 404 returned error can't find the container with id 74a59b76abefddb3b42ae880ccfb8df7538d03ad03fee714c400b67133f82159
Nov 02 01:31:21.275594 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:21.275200    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret\") pod \"global-pull-secret-syncer-8vhjc\" (UID: \"b88b4dfb-1523-4701-9388-781cedab823b\") " pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:21.275594 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:21.275262    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs\") pod \"network-metrics-daemon-gws4h\" (UID: \"841f3428-10c6-4079-a256-cf6e5cd21806\") " pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:21.275594 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:21.275392    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:21.275594 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:21.275456    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs podName:841f3428-10c6-4079-a256-cf6e5cd21806 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:23.27543667 +0000 UTC m=+5.326752443 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs") pod "network-metrics-daemon-gws4h" (UID: "841f3428-10c6-4079-a256-cf6e5cd21806") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:21.275594 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:21.275527    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:21.275594 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:21.275565    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret podName:b88b4dfb-1523-4701-9388-781cedab823b nodeName:}" failed. No retries permitted until 2026-11-02 01:31:23.275552855 +0000 UTC m=+5.326868598 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret") pod "global-pull-secret-syncer-8vhjc" (UID: "b88b4dfb-1523-4701-9388-781cedab823b") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:21.376434 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:21.376394    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9sj2\" (UniqueName: \"kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2\") pod \"network-check-target-gzq27\" (UID: \"6a2f0fa5-cbce-4695-aa38-e8bbeb720280\") " pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:21.377050 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:21.376624    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:31:21.377050 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:21.376649    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:31:21.377050 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:21.376662    2547 projected.go:196] Error preparing data for projected volume kube-api-access-c9sj2 for pod openshift-network-diagnostics/network-check-target-gzq27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:21.377050 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:21.376724    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2 podName:6a2f0fa5-cbce-4695-aa38-e8bbeb720280 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:23.376705672 +0000 UTC m=+5.428021428 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-c9sj2" (UniqueName: "kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2") pod "network-check-target-gzq27" (UID: "6a2f0fa5-cbce-4695-aa38-e8bbeb720280") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:21.496520 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:21.496000    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:21.496520 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:21.496121    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzq27" podUID="6a2f0fa5-cbce-4695-aa38-e8bbeb720280"
Nov 02 01:31:21.496520 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:21.496199    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:21.496520 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:21.496285    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vhjc" podUID="b88b4dfb-1523-4701-9388-781cedab823b"
Nov 02 01:31:21.496520 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:21.496349    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:21.496520 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:21.496420    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gws4h" podUID="841f3428-10c6-4079-a256-cf6e5cd21806"
Nov 02 01:31:21.524821 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:21.524779    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb" event={"ID":"a39f3ef9-2d60-4da6-b69a-9bdfac039833","Type":"ContainerStarted","Data":"69ae9b34a115e17992488ab6bb6a978d66e0bdaf4fdb424715e2eee95957142d"}
Nov 02 01:31:21.526968 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:21.526886    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f297t" event={"ID":"adb8a888-520e-4ef5-819a-ee9331b0ccd9","Type":"ContainerStarted","Data":"74a59b76abefddb3b42ae880ccfb8df7538d03ad03fee714c400b67133f82159"}
Nov 02 01:31:21.528970 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:21.528940    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vh2dn" event={"ID":"902d51a8-2407-4f4b-a5d9-134a40ad956c","Type":"ContainerStarted","Data":"0d06127b2465ebbbb352b6533e80b8f303a389e2b3d2af40b57fc18c3cd0c9ca"}
Nov 02 01:31:21.531262 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:21.531218    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtl6l" event={"ID":"6e02ef8d-7b7b-4de9-a2db-ae32d200a497","Type":"ContainerStarted","Data":"8181c311e1c1935dc496d928f2500bd28e19de558f63de49bf74791c75f5bb03"}
Nov 02 01:31:21.747600 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:21.747555    2547 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 01:26:19 +0000 UTC" deadline="2028-04-28 21:40:14.438054586 +0000 UTC"
Nov 02 01:31:21.747600 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:21.747598    2547 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13052h8m52.690461959s"
Nov 02 01:31:23.293500 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:23.293459    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret\") pod \"global-pull-secret-syncer-8vhjc\" (UID: \"b88b4dfb-1523-4701-9388-781cedab823b\") " pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:23.293986 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:23.293513    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs\") pod \"network-metrics-daemon-gws4h\" (UID: \"841f3428-10c6-4079-a256-cf6e5cd21806\") " pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:23.293986 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:23.293653    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:23.293986 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:23.293715    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs podName:841f3428-10c6-4079-a256-cf6e5cd21806 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:27.293696509 +0000 UTC m=+9.345012246 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs") pod "network-metrics-daemon-gws4h" (UID: "841f3428-10c6-4079-a256-cf6e5cd21806") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:23.293986 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:23.293776    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:23.293986 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:23.293808    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret podName:b88b4dfb-1523-4701-9388-781cedab823b nodeName:}" failed. No retries permitted until 2026-11-02 01:31:27.29379801 +0000 UTC m=+9.345113762 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret") pod "global-pull-secret-syncer-8vhjc" (UID: "b88b4dfb-1523-4701-9388-781cedab823b") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:23.394908 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:23.394515    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9sj2\" (UniqueName: \"kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2\") pod \"network-check-target-gzq27\" (UID: \"6a2f0fa5-cbce-4695-aa38-e8bbeb720280\") " pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:23.394908 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:23.394719    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:31:23.394908 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:23.394738    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:31:23.394908 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:23.394749    2547 projected.go:196] Error preparing data for projected volume kube-api-access-c9sj2 for pod openshift-network-diagnostics/network-check-target-gzq27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:23.394908 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:23.394807    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2 podName:6a2f0fa5-cbce-4695-aa38-e8bbeb720280 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:27.39478842 +0000 UTC m=+9.446104177 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-c9sj2" (UniqueName: "kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2") pod "network-check-target-gzq27" (UID: "6a2f0fa5-cbce-4695-aa38-e8bbeb720280") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:23.495809 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:23.495095    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:23.495809 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:23.495247    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzq27" podUID="6a2f0fa5-cbce-4695-aa38-e8bbeb720280"
Nov 02 01:31:23.495809 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:23.495544    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:23.495809 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:23.495622    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vhjc" podUID="b88b4dfb-1523-4701-9388-781cedab823b"
Nov 02 01:31:23.495809 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:23.495670    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:23.495809 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:23.495764    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gws4h" podUID="841f3428-10c6-4079-a256-cf6e5cd21806"
Nov 02 01:31:25.495082 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:25.494999    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:25.495570 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:25.495145    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gws4h" podUID="841f3428-10c6-4079-a256-cf6e5cd21806"
Nov 02 01:31:25.495570 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:25.495210    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:25.495570 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:25.495300    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzq27" podUID="6a2f0fa5-cbce-4695-aa38-e8bbeb720280"
Nov 02 01:31:25.495570 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:25.495355    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:25.495570 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:25.495417    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vhjc" podUID="b88b4dfb-1523-4701-9388-781cedab823b"
Nov 02 01:31:25.558354 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:25.558311    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal" event={"ID":"5845b2d4146c451ff8e611722d7c3e34","Type":"ContainerStarted","Data":"5037651ea95ef8240a43a9e97ccdcfd1766652cf2ddac2da59cb0331fa23357b"}
Nov 02 01:31:25.563142 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:25.563110    2547 generic.go:356] "Generic (PLEG): container finished" podID="6e02ef8d-7b7b-4de9-a2db-ae32d200a497" containerID="fae0dde6c84757eadc3faeae582c09d223d142a10ed77600932d32219def7661" exitCode=0
Nov 02 01:31:25.563333 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:25.563186    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtl6l" event={"ID":"6e02ef8d-7b7b-4de9-a2db-ae32d200a497","Type":"ContainerDied","Data":"fae0dde6c84757eadc3faeae582c09d223d142a10ed77600932d32219def7661"}
Nov 02 01:31:25.567804 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:25.566777    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ft5bg" event={"ID":"65f63e43-859a-4d3a-8393-99724782596b","Type":"ContainerStarted","Data":"4a9760a59af639b2f1f33b07c2366fdc7567fcaab44fd544e83aed06e1bbcba1"}
Nov 02 01:31:25.570826 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:25.570365    2547 generic.go:356] "Generic (PLEG): container finished" podID="efa1d9dffd42c4f4f5c559910849b4c9" containerID="42d7888ebbaf80be1529103f2845105888ef4b5e0f45756620fe6cd5dab8d2a0" exitCode=0
Nov 02 01:31:25.570826 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:25.570402    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal" event={"ID":"efa1d9dffd42c4f4f5c559910849b4c9","Type":"ContainerDied","Data":"42d7888ebbaf80be1529103f2845105888ef4b5e0f45756620fe6cd5dab8d2a0"}
Nov 02 01:31:25.671856 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:25.671788    2547 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal" podStartSLOduration=7.671770809 podStartE2EDuration="7.671770809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:25.671386675 +0000 UTC m=+7.722702448" watchObservedRunningTime="2026-11-02 01:31:25.671770809 +0000 UTC m=+7.723086570"
Nov 02 01:31:26.577248 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:26.577149    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c" event={"ID":"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28","Type":"ContainerStarted","Data":"a8d6b4c964ca5b4ac0459e9b677ff479da26c7cd47138a0e0e4f1044e7f4dd50"}
Nov 02 01:31:26.583433 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:26.583393    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal" event={"ID":"efa1d9dffd42c4f4f5c559910849b4c9","Type":"ContainerStarted","Data":"038c8456a8da65c5a13a62eee8ba07498f12e36406f999e41c8498bf2c09f489"}
Nov 02 01:31:26.586384 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:26.586356    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7xs66" event={"ID":"046dd898-c779-4ce2-84ec-dc91afb8323f","Type":"ContainerStarted","Data":"4834e7665f40998be83a193c9d46942c1d13152a0921aaa7e4b8fe6535cd23f6"}
Nov 02 01:31:26.698410 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:26.697432    2547 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ft5bg" podStartSLOduration=3.850056618 podStartE2EDuration="8.697414704s" totalImagesPullingTime="4.847358086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:26.016156921 +0000 UTC m=+8.067472680" watchObservedRunningTime="2026-11-02 01:31:26.697414704 +0000 UTC m=+8.748730464"
Nov 02 01:31:26.811574 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:26.811516    2547 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal" podStartSLOduration=8.811496143 podStartE2EDuration="8.811496143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:26.698445579 +0000 UTC m=+8.749761338" watchObservedRunningTime="2026-11-02 01:31:26.811496143 +0000 UTC m=+8.862811904"
Nov 02 01:31:26.843578 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:26.843510    2547 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ft5bg"
Nov 02 01:31:26.844138 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:26.844109    2547 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ft5bg"
Nov 02 01:31:26.957132 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:26.957079    2547 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7xs66" podStartSLOduration=3.039500508 podStartE2EDuration="8.95706039s" totalImagesPullingTime="5.917559882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:26.812277449 +0000 UTC m=+8.863593204" watchObservedRunningTime="2026-11-02 01:31:26.95706039 +0000 UTC m=+9.008376149"
Nov 02 01:31:27.329426 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:27.329352    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret\") pod \"global-pull-secret-syncer-8vhjc\" (UID: \"b88b4dfb-1523-4701-9388-781cedab823b\") " pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:27.329426 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:27.329392    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs\") pod \"network-metrics-daemon-gws4h\" (UID: \"841f3428-10c6-4079-a256-cf6e5cd21806\") " pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:27.329630 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:27.329504    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:27.329630 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:27.329553    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs podName:841f3428-10c6-4079-a256-cf6e5cd21806 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:35.329540046 +0000 UTC m=+17.380855783 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs") pod "network-metrics-daemon-gws4h" (UID: "841f3428-10c6-4079-a256-cf6e5cd21806") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:27.329630 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:27.329610    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:27.329630 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:27.329631    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret podName:b88b4dfb-1523-4701-9388-781cedab823b nodeName:}" failed. No retries permitted until 2026-11-02 01:31:35.329625218 +0000 UTC m=+17.380940953 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret") pod "global-pull-secret-syncer-8vhjc" (UID: "b88b4dfb-1523-4701-9388-781cedab823b") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:27.430143 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:27.430112    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9sj2\" (UniqueName: \"kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2\") pod \"network-check-target-gzq27\" (UID: \"6a2f0fa5-cbce-4695-aa38-e8bbeb720280\") " pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:27.430347 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:27.430325    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:31:27.430433 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:27.430387    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:31:27.430433 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:27.430398    2547 projected.go:196] Error preparing data for projected volume kube-api-access-c9sj2 for pod openshift-network-diagnostics/network-check-target-gzq27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:27.430535 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:27.430453    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2 podName:6a2f0fa5-cbce-4695-aa38-e8bbeb720280 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:35.43043951 +0000 UTC m=+17.481755246 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-c9sj2" (UniqueName: "kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2") pod "network-check-target-gzq27" (UID: "6a2f0fa5-cbce-4695-aa38-e8bbeb720280") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:27.495693 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:27.495656    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:27.495850 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:27.495790    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzq27" podUID="6a2f0fa5-cbce-4695-aa38-e8bbeb720280"
Nov 02 01:31:27.495907 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:27.495852    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:27.495969 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:27.495918    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vhjc" podUID="b88b4dfb-1523-4701-9388-781cedab823b"
Nov 02 01:31:27.496018 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:27.495968    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:27.496076 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:27.496039    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gws4h" podUID="841f3428-10c6-4079-a256-cf6e5cd21806"
Nov 02 01:31:27.590587 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:27.590484    2547 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ft5bg"
Nov 02 01:31:27.591252 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:27.591209    2547 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ft5bg"
Nov 02 01:31:28.440419 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:28.440384    2547 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 01:31:28.596555 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:28.595694    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7tt8p" event={"ID":"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20","Type":"ContainerStarted","Data":"933aefe2f0f3ad1377a7e417e5dae1268bc12ae8515b51bc110c258fa206c354"}
Nov 02 01:31:28.605211 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:28.605180    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c" event={"ID":"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28","Type":"ContainerStarted","Data":"939cc77231bbb2027a62e9ce136322a6b50f2697e581b219f282d8b562efeb05"}
Nov 02 01:31:28.713735 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:28.713650    2547 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-7tt8p" podStartSLOduration=2.548182828 podStartE2EDuration="10.71363678s" totalImagesPullingTime="8.165453952s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:28.713519243 +0000 UTC m=+10.764835000" watchObservedRunningTime="2026-11-02 01:31:28.71363678 +0000 UTC m=+10.764952538"
Nov 02 01:31:29.423056 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:29.422918    2547 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T01:31:28.440401023Z","UUID":"734349bb-62f9-4038-8a0d-e49ba3342dec","Handler":null,"Name":"","Endpoint":""}
Nov 02 01:31:29.424660 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:29.424630    2547 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 01:31:29.424660 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:29.424657    2547 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 01:31:29.495058 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:29.495029    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:29.495179 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:29.495144    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzq27" podUID="6a2f0fa5-cbce-4695-aa38-e8bbeb720280"
Nov 02 01:31:29.495276 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:29.495205    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:29.495338 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:29.495318    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vhjc" podUID="b88b4dfb-1523-4701-9388-781cedab823b"
Nov 02 01:31:29.495394 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:29.495378    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:29.495475 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:29.495427    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gws4h" podUID="841f3428-10c6-4079-a256-cf6e5cd21806"
Nov 02 01:31:29.608931 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:29.608655    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bd696" event={"ID":"9f798cfa-1160-4da5-859c-47d81c07f969","Type":"ContainerStarted","Data":"d5090354b9203e40f66392f807b7d320d1f77a2292412c5f3659527fda756e78"}
Nov 02 01:31:29.612361 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:29.612284    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtl6l" event={"ID":"6e02ef8d-7b7b-4de9-a2db-ae32d200a497","Type":"ContainerStarted","Data":"72889d8bba7bc213e6beae588070bf9b89130aebab136e743923e9260d993cf5"}
Nov 02 01:31:29.722883 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:29.722532    2547 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-bd696" podStartSLOduration=3.20253059 podStartE2EDuration="11.722513697s" totalImagesPullingTime="8.519983107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:29.721890747 +0000 UTC m=+11.773206516" watchObservedRunningTime="2026-11-02 01:31:29.722513697 +0000 UTC m=+11.773829454"
Nov 02 01:31:30.617908 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:30.617865    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c" event={"ID":"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28","Type":"ContainerStarted","Data":"2c34c97bf786e3999d19dc6fe33018059030234085335c997a3dc13cfae38c49"}
Nov 02 01:31:30.620175 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:30.620126    2547 generic.go:356] "Generic (PLEG): container finished" podID="6e02ef8d-7b7b-4de9-a2db-ae32d200a497" containerID="72889d8bba7bc213e6beae588070bf9b89130aebab136e743923e9260d993cf5" exitCode=0
Nov 02 01:31:30.620316 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:30.620284    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtl6l" event={"ID":"6e02ef8d-7b7b-4de9-a2db-ae32d200a497","Type":"ContainerDied","Data":"72889d8bba7bc213e6beae588070bf9b89130aebab136e743923e9260d993cf5"}
Nov 02 01:31:30.735553 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:30.735473    2547 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c" podStartSLOduration=3.172077162 podStartE2EDuration="12.735444966s" totalImagesPullingTime="9.563367804s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:31:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:30.735148179 +0000 UTC m=+12.786463938" watchObservedRunningTime="2026-11-02 01:31:30.735444966 +0000 UTC m=+12.786760724"
Nov 02 01:31:31.495474 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:31.495386    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:31.495474 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:31.495464    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:31.495672 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:31.495477    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:31.495672 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:31.495568    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzq27" podUID="6a2f0fa5-cbce-4695-aa38-e8bbeb720280"
Nov 02 01:31:31.495672 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:31.495622    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gws4h" podUID="841f3428-10c6-4079-a256-cf6e5cd21806"
Nov 02 01:31:31.495794 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:31.495719    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vhjc" podUID="b88b4dfb-1523-4701-9388-781cedab823b"
Nov 02 01:31:31.624619 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:31.624582    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f297t" event={"ID":"adb8a888-520e-4ef5-819a-ee9331b0ccd9","Type":"ContainerStarted","Data":"e248b69484968a97618c6b9c70c8ed06be69547884ad396a321673871d899086"}
Nov 02 01:31:31.626393 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:31.626362    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vh2dn" event={"ID":"902d51a8-2407-4f4b-a5d9-134a40ad956c","Type":"ContainerStarted","Data":"1a7b2ceb34a5f99d033ba8b1ffb6c5ac4c2b0cdf4323c3d93dc7d1ed7d0f0bb2"}
Nov 02 01:31:31.738091 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:31.738029    2547 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f297t" podStartSLOduration=5.465481073 podStartE2EDuration="13.738014645s" totalImagesPullingTime="8.272533572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:31.737391148 +0000 UTC m=+13.788706909" watchObservedRunningTime="2026-11-02 01:31:31.738014645 +0000 UTC m=+13.789330405"
Nov 02 01:31:31.853912 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:31.853813    2547 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vh2dn" podStartSLOduration=3.525722306 podStartE2EDuration="13.853795589s" totalImagesPullingTime="10.328073283s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:31.853137029 +0000 UTC m=+13.904452788" watchObservedRunningTime="2026-11-02 01:31:31.853795589 +0000 UTC m=+13.905111346"
Nov 02 01:31:32.630992 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:32.630912    2547 generic.go:356] "Generic (PLEG): container finished" podID="6e02ef8d-7b7b-4de9-a2db-ae32d200a497" containerID="ceb685ab5c72108b8d0ff2a402a4a6e11dee59ee926b81ce2cadb422d81f2683" exitCode=0
Nov 02 01:31:32.631691 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:32.630983    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtl6l" event={"ID":"6e02ef8d-7b7b-4de9-a2db-ae32d200a497","Type":"ContainerDied","Data":"ceb685ab5c72108b8d0ff2a402a4a6e11dee59ee926b81ce2cadb422d81f2683"}
Nov 02 01:31:33.495592 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:33.495555    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:33.495759 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:33.495644    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:33.495759 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:33.495742    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:33.495839 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:33.495750    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzq27" podUID="6a2f0fa5-cbce-4695-aa38-e8bbeb720280"
Nov 02 01:31:33.495871 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:33.495835    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vhjc" podUID="b88b4dfb-1523-4701-9388-781cedab823b"
Nov 02 01:31:33.495938 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:33.495917    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gws4h" podUID="841f3428-10c6-4079-a256-cf6e5cd21806"
Nov 02 01:31:34.638470 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:34.638428    2547 generic.go:356] "Generic (PLEG): container finished" podID="6e02ef8d-7b7b-4de9-a2db-ae32d200a497" containerID="ac49da3ce13eeed309476d58183bb5b3b6287c29fa03dffb5aee884d03028e1f" exitCode=0
Nov 02 01:31:34.638470 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:34.638471    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtl6l" event={"ID":"6e02ef8d-7b7b-4de9-a2db-ae32d200a497","Type":"ContainerDied","Data":"ac49da3ce13eeed309476d58183bb5b3b6287c29fa03dffb5aee884d03028e1f"}
Nov 02 01:31:35.396602 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:35.396568    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret\") pod \"global-pull-secret-syncer-8vhjc\" (UID: \"b88b4dfb-1523-4701-9388-781cedab823b\") " pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:35.396602 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:35.396608    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs\") pod \"network-metrics-daemon-gws4h\" (UID: \"841f3428-10c6-4079-a256-cf6e5cd21806\") " pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:35.396865 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:35.396739    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:35.396865 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:35.396744    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:35.396865 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:35.396802    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs podName:841f3428-10c6-4079-a256-cf6e5cd21806 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:51.396785928 +0000 UTC m=+33.448101678 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs") pod "network-metrics-daemon-gws4h" (UID: "841f3428-10c6-4079-a256-cf6e5cd21806") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:35.396865 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:35.396822    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret podName:b88b4dfb-1523-4701-9388-781cedab823b nodeName:}" failed. No retries permitted until 2026-11-02 01:31:51.396811678 +0000 UTC m=+33.448127413 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret") pod "global-pull-secret-syncer-8vhjc" (UID: "b88b4dfb-1523-4701-9388-781cedab823b") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:35.495429 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:35.495393    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:35.495607 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:35.495432    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:35.495607 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:35.495440    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:35.495607 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:35.495535    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gws4h" podUID="841f3428-10c6-4079-a256-cf6e5cd21806"
Nov 02 01:31:35.495755 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:35.495600    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzq27" podUID="6a2f0fa5-cbce-4695-aa38-e8bbeb720280"
Nov 02 01:31:35.495755 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:35.495704    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vhjc" podUID="b88b4dfb-1523-4701-9388-781cedab823b"
Nov 02 01:31:35.497589 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:35.497560    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9sj2\" (UniqueName: \"kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2\") pod \"network-check-target-gzq27\" (UID: \"6a2f0fa5-cbce-4695-aa38-e8bbeb720280\") " pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:35.497721 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:35.497684    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:31:35.497721 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:35.497702    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:31:35.497721 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:35.497711    2547 projected.go:196] Error preparing data for projected volume kube-api-access-c9sj2 for pod openshift-network-diagnostics/network-check-target-gzq27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:35.497834 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:35.497752    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2 podName:6a2f0fa5-cbce-4695-aa38-e8bbeb720280 nodeName:}" failed. No retries permitted until 2026-11-02 01:31:51.497739021 +0000 UTC m=+33.549054772 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-c9sj2" (UniqueName: "kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2") pod "network-check-target-gzq27" (UID: "6a2f0fa5-cbce-4695-aa38-e8bbeb720280") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:36.646720 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:36.646528    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb" event={"ID":"a39f3ef9-2d60-4da6-b69a-9bdfac039833","Type":"ContainerStarted","Data":"8f10fdbde73b37b985b2c2d1d2a789e5b7cd2d88298216f779a8b84e4b429025"}
Nov 02 01:31:36.647166 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:36.646765    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb" event={"ID":"a39f3ef9-2d60-4da6-b69a-9bdfac039833","Type":"ContainerStarted","Data":"67e175d755c811453015e6c2bbb2d4e74c5b22a2418831d4cc7e7d14eba80722"}
Nov 02 01:31:36.647166 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:36.646801    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb" event={"ID":"a39f3ef9-2d60-4da6-b69a-9bdfac039833","Type":"ContainerStarted","Data":"353272943d86855e2186ee964dd817fb1a1bd74e351e7f2b37c5d9fe5a85f023"}
Nov 02 01:31:36.647166 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:36.646838    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb" event={"ID":"a39f3ef9-2d60-4da6-b69a-9bdfac039833","Type":"ContainerStarted","Data":"4c4cb375a8f3d81a2d96acee5419ff2c8afa977d7d5807852628d1b121ccea1d"}
Nov 02 01:31:36.647166 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:36.646886    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb" event={"ID":"a39f3ef9-2d60-4da6-b69a-9bdfac039833","Type":"ContainerStarted","Data":"37634d00831f98ad4ebbefe3432b18b988ff0843e64bad84b7ce6f57f46676b0"}
Nov 02 01:31:36.647166 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:36.646920    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb" event={"ID":"a39f3ef9-2d60-4da6-b69a-9bdfac039833","Type":"ContainerStarted","Data":"badd9b67729adae134ac60116cd4c2fc2bab66364dae1d5229e32b44ac4f5cff"}
Nov 02 01:31:37.495720 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:37.495689    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:37.495720 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:37.495722    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:37.495930 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:37.495734    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:37.495930 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:37.495840    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vhjc" podUID="b88b4dfb-1523-4701-9388-781cedab823b"
Nov 02 01:31:37.495986 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:37.495953    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzq27" podUID="6a2f0fa5-cbce-4695-aa38-e8bbeb720280"
Nov 02 01:31:37.496056 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:37.496041    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gws4h" podUID="841f3428-10c6-4079-a256-cf6e5cd21806"
Nov 02 01:31:37.651021 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:37.650991    2547 generic.go:356] "Generic (PLEG): container finished" podID="6e02ef8d-7b7b-4de9-a2db-ae32d200a497" containerID="c2e715d07decb52d5aeeae4af4e7e221f330030008468667498ddd1d21a9cdde" exitCode=0
Nov 02 01:31:37.651021 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:37.651026    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtl6l" event={"ID":"6e02ef8d-7b7b-4de9-a2db-ae32d200a497","Type":"ContainerDied","Data":"c2e715d07decb52d5aeeae4af4e7e221f330030008468667498ddd1d21a9cdde"}
Nov 02 01:31:38.656363 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:38.656332    2547 generic.go:356] "Generic (PLEG): container finished" podID="6e02ef8d-7b7b-4de9-a2db-ae32d200a497" containerID="183dc1c619fb4a32b6c6979e3f9c2415ffaf979f6f0da62cb11b43c26314ae35" exitCode=0
Nov 02 01:31:38.656965 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:38.656385    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtl6l" event={"ID":"6e02ef8d-7b7b-4de9-a2db-ae32d200a497","Type":"ContainerDied","Data":"183dc1c619fb4a32b6c6979e3f9c2415ffaf979f6f0da62cb11b43c26314ae35"}
Nov 02 01:31:38.659957 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:38.659938    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb" event={"ID":"a39f3ef9-2d60-4da6-b69a-9bdfac039833","Type":"ContainerStarted","Data":"3b01273f376543f11e052d7a0632df2e5e7d2ba970627836a1270579c06cb7eb"}
Nov 02 01:31:39.495895 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:39.495863    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:39.495895 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:39.495896    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:39.495895 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:39.495893    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:39.496126 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:39.495981    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gws4h" podUID="841f3428-10c6-4079-a256-cf6e5cd21806"
Nov 02 01:31:39.496126 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:39.496091    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzq27" podUID="6a2f0fa5-cbce-4695-aa38-e8bbeb720280"
Nov 02 01:31:39.496183 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:39.496150    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vhjc" podUID="b88b4dfb-1523-4701-9388-781cedab823b"
Nov 02 01:31:39.665144 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:39.665111    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtl6l" event={"ID":"6e02ef8d-7b7b-4de9-a2db-ae32d200a497","Type":"ContainerStarted","Data":"6286fff2637391382c38f2fa158e647b30a928185437778fd20685c52ab121d1"}
Nov 02 01:31:39.837136 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:39.837041    2547 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jtl6l" podStartSLOduration=9.528388861 podStartE2EDuration="21.837027596s" totalImagesPullingTime="12.308638735s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:31:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:39.836642604 +0000 UTC m=+21.887958361" watchObservedRunningTime="2026-11-02 01:31:39.837027596 +0000 UTC m=+21.888343354"
Nov 02 01:31:41.495393 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:41.495361    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:41.496033 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:41.495394    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:41.496033 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:41.495478    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gws4h" podUID="841f3428-10c6-4079-a256-cf6e5cd21806"
Nov 02 01:31:41.496033 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:41.495517    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:41.496033 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:41.495603    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzq27" podUID="6a2f0fa5-cbce-4695-aa38-e8bbeb720280"
Nov 02 01:31:41.496033 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:41.495637    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vhjc" podUID="b88b4dfb-1523-4701-9388-781cedab823b"
Nov 02 01:31:41.672790 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:41.672759    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb" event={"ID":"a39f3ef9-2d60-4da6-b69a-9bdfac039833","Type":"ContainerStarted","Data":"3a596376df301799f2d099f5d9bcbcb35aebd870cab5d00cedf90cc2e7123726"}
Nov 02 01:31:41.675587 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:41.673587    2547 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:41.675587 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:41.673846    2547 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:41.691970 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:41.691949    2547 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:41.948204 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:41.948110    2547 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb" podStartSLOduration=8.57734488 podStartE2EDuration="23.948094577s" totalImagesPullingTime="15.370749697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:41.927769516 +0000 UTC m=+23.979085275" watchObservedRunningTime="2026-11-02 01:31:41.948094577 +0000 UTC m=+23.999410359"
Nov 02 01:31:42.675937 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:42.675760    2547 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:42.691203 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:42.691182    2547 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:31:43.432813 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:43.432738    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8vhjc"]
Nov 02 01:31:43.433005 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:43.432860    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:43.433005 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:43.432956    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vhjc" podUID="b88b4dfb-1523-4701-9388-781cedab823b"
Nov 02 01:31:43.436014 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:43.435780    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gzq27"]
Nov 02 01:31:43.436014 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:43.435884    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:43.436014 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:43.435975    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzq27" podUID="6a2f0fa5-cbce-4695-aa38-e8bbeb720280"
Nov 02 01:31:43.436522 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:43.436495    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gws4h"]
Nov 02 01:31:43.436633 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:43.436601    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:43.436697 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:43.436677    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gws4h" podUID="841f3428-10c6-4079-a256-cf6e5cd21806"
Nov 02 01:31:44.495284 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:44.495252    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:44.495680 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:44.495355    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzq27" podUID="6a2f0fa5-cbce-4695-aa38-e8bbeb720280"
Nov 02 01:31:45.495887 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:45.495854    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:45.495887 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:45.495887    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:45.496438 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:45.496018    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gws4h" podUID="841f3428-10c6-4079-a256-cf6e5cd21806"
Nov 02 01:31:45.496438 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:45.496100    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vhjc" podUID="b88b4dfb-1523-4701-9388-781cedab823b"
Nov 02 01:31:46.495595 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:46.495565    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:46.495779 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:46.495671    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzq27" podUID="6a2f0fa5-cbce-4695-aa38-e8bbeb720280"
Nov 02 01:31:47.495122 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:47.495092    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:47.495581 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:47.495238    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vhjc" podUID="b88b4dfb-1523-4701-9388-781cedab823b"
Nov 02 01:31:47.495581 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:47.495245    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:47.495581 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:47.495354    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gws4h" podUID="841f3428-10c6-4079-a256-cf6e5cd21806"
Nov 02 01:31:47.932248 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:47.932149    2547 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ft5bg_65f63e43-859a-4d3a-8393-99724782596b/konnectivity-agent/0.log" limit=1024
Nov 02 01:31:48.496357 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:48.496329    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:48.496824 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:48.496424    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzq27" podUID="6a2f0fa5-cbce-4695-aa38-e8bbeb720280"
Nov 02 01:31:49.495103 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:49.495067    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:49.495103 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:49.495102    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:49.495348 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:49.495178    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vhjc" podUID="b88b4dfb-1523-4701-9388-781cedab823b"
Nov 02 01:31:49.495348 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:49.495268    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gws4h" podUID="841f3428-10c6-4079-a256-cf6e5cd21806"
Nov 02 01:31:50.495268 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:50.495214    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:50.495666 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:50.495359    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzq27" podUID="6a2f0fa5-cbce-4695-aa38-e8bbeb720280"
Nov 02 01:31:51.406876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:51.406837    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret\") pod \"global-pull-secret-syncer-8vhjc\" (UID: \"b88b4dfb-1523-4701-9388-781cedab823b\") " pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:51.406876 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:51.406878    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs\") pod \"network-metrics-daemon-gws4h\" (UID: \"841f3428-10c6-4079-a256-cf6e5cd21806\") " pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:51.407128 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:51.406969    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:51.407128 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:51.406974    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:51.407128 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:51.407015    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs podName:841f3428-10c6-4079-a256-cf6e5cd21806 nodeName:}" failed. No retries permitted until 2026-11-02 01:32:23.407002972 +0000 UTC m=+65.458318709 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs") pod "network-metrics-daemon-gws4h" (UID: "841f3428-10c6-4079-a256-cf6e5cd21806") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:31:51.407128 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:51.407028    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret podName:b88b4dfb-1523-4701-9388-781cedab823b nodeName:}" failed. No retries permitted until 2026-11-02 01:32:23.407021461 +0000 UTC m=+65.458337197 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret") pod "global-pull-secret-syncer-8vhjc" (UID: "b88b4dfb-1523-4701-9388-781cedab823b") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:31:51.495295 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:51.495259    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:51.495715 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:51.495308    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:51.495715 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:51.495486    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vhjc" podUID="b88b4dfb-1523-4701-9388-781cedab823b"
Nov 02 01:31:51.495715 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:51.495582    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gws4h" podUID="841f3428-10c6-4079-a256-cf6e5cd21806"
Nov 02 01:31:51.507416 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:51.507394    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9sj2\" (UniqueName: \"kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2\") pod \"network-check-target-gzq27\" (UID: \"6a2f0fa5-cbce-4695-aa38-e8bbeb720280\") " pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:51.507548 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:51.507534    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:31:51.507592 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:51.507554    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:31:51.507592 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:51.507563    2547 projected.go:196] Error preparing data for projected volume kube-api-access-c9sj2 for pod openshift-network-diagnostics/network-check-target-gzq27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:51.507663 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:51.507601    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2 podName:6a2f0fa5-cbce-4695-aa38-e8bbeb720280 nodeName:}" failed. No retries permitted until 2026-11-02 01:32:23.507589866 +0000 UTC m=+65.558905618 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-c9sj2" (UniqueName: "kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2") pod "network-check-target-gzq27" (UID: "6a2f0fa5-cbce-4695-aa38-e8bbeb720280") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:31:52.495516 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:52.495478    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:52.495986 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:52.495606    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzq27" podUID="6a2f0fa5-cbce-4695-aa38-e8bbeb720280"
Nov 02 01:31:53.495657 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:53.495625    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:53.496075 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:53.495738    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gws4h" podUID="841f3428-10c6-4079-a256-cf6e5cd21806"
Nov 02 01:31:53.496075 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:53.495789    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:53.496075 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:53.495876    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vhjc" podUID="b88b4dfb-1523-4701-9388-781cedab823b"
Nov 02 01:31:53.516477 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:53.516454    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bd696_9f798cfa-1160-4da5-859c-47d81c07f969/dns-node-resolver/0.log"
Nov 02 01:31:54.495890 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:54.495863    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:54.496297 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:54.495962    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzq27" podUID="6a2f0fa5-cbce-4695-aa38-e8bbeb720280"
Nov 02 01:31:54.514085 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:54.514062    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7xs66_046dd898-c779-4ce2-84ec-dc91afb8323f/node-ca/0.log"
Nov 02 01:31:55.495794 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:55.495763    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:55.495794 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:55.495797    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:55.496340 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:55.495869    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gws4h" podUID="841f3428-10c6-4079-a256-cf6e5cd21806"
Nov 02 01:31:55.496340 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:55.496000    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vhjc" podUID="b88b4dfb-1523-4701-9388-781cedab823b"
Nov 02 01:31:56.495539 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:56.495503    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:56.495697 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:56.495610    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzq27" podUID="6a2f0fa5-cbce-4695-aa38-e8bbeb720280"
Nov 02 01:31:57.495147 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:57.495109    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:57.495644 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:57.495240    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gws4h" podUID="841f3428-10c6-4079-a256-cf6e5cd21806"
Nov 02 01:31:57.495644 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:57.495283    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:57.495644 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:57.495405    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vhjc" podUID="b88b4dfb-1523-4701-9388-781cedab823b"
Nov 02 01:31:58.496402 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:58.496373    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:31:58.496785 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:58.496452    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzq27" podUID="6a2f0fa5-cbce-4695-aa38-e8bbeb720280"
Nov 02 01:31:59.495974 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:59.495940    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:31:59.495974 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:31:59.495966    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:31:59.496180 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:59.496050    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gws4h" podUID="841f3428-10c6-4079-a256-cf6e5cd21806"
Nov 02 01:31:59.496216 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:31:59.496174    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vhjc" podUID="b88b4dfb-1523-4701-9388-781cedab823b"
Nov 02 01:32:00.495697 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:00.495665    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:32:00.496214 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:32:00.495795    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzq27" podUID="6a2f0fa5-cbce-4695-aa38-e8bbeb720280"
Nov 02 01:32:01.495198 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:01.495168    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:32:01.495373 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:32:01.495292    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gws4h" podUID="841f3428-10c6-4079-a256-cf6e5cd21806"
Nov 02 01:32:01.495373 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:01.495360    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:32:01.495464 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:32:01.495446    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vhjc" podUID="b88b4dfb-1523-4701-9388-781cedab823b"
Nov 02 01:32:02.498401 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:02.498375    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:32:02.498774 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:32:02.498470    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzq27" podUID="6a2f0fa5-cbce-4695-aa38-e8bbeb720280"
Nov 02 01:32:03.495603 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:03.495566    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:32:03.495793 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:32:03.495680    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gws4h" podUID="841f3428-10c6-4079-a256-cf6e5cd21806"
Nov 02 01:32:03.495793 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:03.495739    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:32:03.495919 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:32:03.495856    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vhjc" podUID="b88b4dfb-1523-4701-9388-781cedab823b"
Nov 02 01:32:04.498423 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:04.498398    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:32:04.498782 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:32:04.498487    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzq27" podUID="6a2f0fa5-cbce-4695-aa38-e8bbeb720280"
Nov 02 01:32:05.495422 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:05.495391    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:32:05.495581 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:05.495432    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:32:05.495581 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:32:05.495509    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gws4h" podUID="841f3428-10c6-4079-a256-cf6e5cd21806"
Nov 02 01:32:05.495581 ip-10-0-141-68 kubenswrapper[2547]: E1102 01:32:05.495572    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vhjc" podUID="b88b4dfb-1523-4701-9388-781cedab823b"
Nov 02 01:32:06.252911 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.252880    2547 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeReady"
Nov 02 01:32:06.253448 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.253019    2547 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 01:32:06.367429 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.367396    2547 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dbzkb"]
Nov 02 01:32:06.368992 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.368973    2547 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-r4njj"]
Nov 02 01:32:06.369111 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.369093    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dbzkb"
Nov 02 01:32:06.370637 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.370611    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:06.374852 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.374831    2547 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-s275r"]
Nov 02 01:32:06.377311 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.377295    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s275r"
Nov 02 01:32:06.416163 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.416134    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dbzkb"]
Nov 02 01:32:06.422950 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.422924    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 01:32:06.423063 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.422926    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 01:32:06.423063 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.422933    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 01:32:06.423063 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.423043    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 01:32:06.423347 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.423331    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 01:32:06.423347 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.423338    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6csd8\""
Nov 02 01:32:06.423463 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.423392    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 01:32:06.423463 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.423414    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qn9km\""
Nov 02 01:32:06.428874 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.428854    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r4njj"]
Nov 02 01:32:06.443539 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.443517    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s275r"]
Nov 02 01:32:06.495781 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.495747    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:32:06.510848 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.510800    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-crio-socket\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:06.510848 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.510828    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26fmn\" (UniqueName: \"kubernetes.io/projected/f857cdde-184c-436f-a8bd-3de120963268-kube-api-access-26fmn\") pod \"ingress-canary-s275r\" (UID: \"f857cdde-184c-436f-a8bd-3de120963268\") " pod="openshift-ingress-canary/ingress-canary-s275r"
Nov 02 01:32:06.510848 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.510847    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28zhc\" (UniqueName: \"kubernetes.io/projected/91a74a01-d2c7-42bd-9ead-8bdc48398e28-kube-api-access-28zhc\") pod \"dns-default-dbzkb\" (UID: \"91a74a01-d2c7-42bd-9ead-8bdc48398e28\") " pod="openshift-dns/dns-default-dbzkb"
Nov 02 01:32:06.511010 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.510867    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt4vp\" (UniqueName: \"kubernetes.io/projected/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-kube-api-access-jt4vp\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:06.511010 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.510944    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:06.511010 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.511005    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/91a74a01-d2c7-42bd-9ead-8bdc48398e28-config-volume\") pod \"dns-default-dbzkb\" (UID: \"91a74a01-d2c7-42bd-9ead-8bdc48398e28\") " pod="openshift-dns/dns-default-dbzkb"
Nov 02 01:32:06.511099 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.511026    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91a74a01-d2c7-42bd-9ead-8bdc48398e28-metrics-tls\") pod \"dns-default-dbzkb\" (UID: \"91a74a01-d2c7-42bd-9ead-8bdc48398e28\") " pod="openshift-dns/dns-default-dbzkb"
Nov 02 01:32:06.511099 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.511052    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f857cdde-184c-436f-a8bd-3de120963268-cert\") pod \"ingress-canary-s275r\" (UID: \"f857cdde-184c-436f-a8bd-3de120963268\") " pod="openshift-ingress-canary/ingress-canary-s275r"
Nov 02 01:32:06.511158 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.511102    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-data-volume\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:06.511158 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.511132    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91a74a01-d2c7-42bd-9ead-8bdc48398e28-tmp-dir\") pod \"dns-default-dbzkb\" (UID: \"91a74a01-d2c7-42bd-9ead-8bdc48398e28\") " pod="openshift-dns/dns-default-dbzkb"
Nov 02 01:32:06.511158 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.511150    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:06.523711 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.523684    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 01:32:06.523792 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.523738    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 01:32:06.523792 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.523754    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xjcn9\""
Nov 02 01:32:06.523792 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.523788    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 01:32:06.547446 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.547420    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 01:32:06.547525 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.547484    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 01:32:06.547651 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.547638    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-42sf2\""
Nov 02 01:32:06.611732 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.611708    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91a74a01-d2c7-42bd-9ead-8bdc48398e28-tmp-dir\") pod \"dns-default-dbzkb\" (UID: \"91a74a01-d2c7-42bd-9ead-8bdc48398e28\") " pod="openshift-dns/dns-default-dbzkb"
Nov 02 01:32:06.611877 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.611739    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:06.611877 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.611773    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-crio-socket\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:06.611877 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.611791    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26fmn\" (UniqueName: \"kubernetes.io/projected/f857cdde-184c-436f-a8bd-3de120963268-kube-api-access-26fmn\") pod \"ingress-canary-s275r\" (UID: \"f857cdde-184c-436f-a8bd-3de120963268\") " pod="openshift-ingress-canary/ingress-canary-s275r"
Nov 02 01:32:06.611988 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.611940    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-crio-socket\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:06.612049 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.612031    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28zhc\" (UniqueName: \"kubernetes.io/projected/91a74a01-d2c7-42bd-9ead-8bdc48398e28-kube-api-access-28zhc\") pod \"dns-default-dbzkb\" (UID: \"91a74a01-d2c7-42bd-9ead-8bdc48398e28\") " pod="openshift-dns/dns-default-dbzkb"
Nov 02 01:32:06.612083 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.612062    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt4vp\" (UniqueName: \"kubernetes.io/projected/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-kube-api-access-jt4vp\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:06.612122 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.612075    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91a74a01-d2c7-42bd-9ead-8bdc48398e28-tmp-dir\") pod \"dns-default-dbzkb\" (UID: \"91a74a01-d2c7-42bd-9ead-8bdc48398e28\") " pod="openshift-dns/dns-default-dbzkb"
Nov 02 01:32:06.612122 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.612083    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:06.612195 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.612144    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/91a74a01-d2c7-42bd-9ead-8bdc48398e28-config-volume\") pod \"dns-default-dbzkb\" (UID: \"91a74a01-d2c7-42bd-9ead-8bdc48398e28\") " pod="openshift-dns/dns-default-dbzkb"
Nov 02 01:32:06.612195 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.612178    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91a74a01-d2c7-42bd-9ead-8bdc48398e28-metrics-tls\") pod \"dns-default-dbzkb\" (UID: \"91a74a01-d2c7-42bd-9ead-8bdc48398e28\") " pod="openshift-dns/dns-default-dbzkb"
Nov 02 01:32:06.612312 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.612215    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f857cdde-184c-436f-a8bd-3de120963268-cert\") pod \"ingress-canary-s275r\" (UID: \"f857cdde-184c-436f-a8bd-3de120963268\") " pod="openshift-ingress-canary/ingress-canary-s275r"
Nov 02 01:32:06.612312 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.612276    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-data-volume\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:06.612582 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.612558    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-data-volume\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:06.612582 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.612561    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:06.612731 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.612681    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/91a74a01-d2c7-42bd-9ead-8bdc48398e28-config-volume\") pod \"dns-default-dbzkb\" (UID: \"91a74a01-d2c7-42bd-9ead-8bdc48398e28\") " pod="openshift-dns/dns-default-dbzkb"
Nov 02 01:32:06.615811 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.615789    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91a74a01-d2c7-42bd-9ead-8bdc48398e28-metrics-tls\") pod \"dns-default-dbzkb\" (UID: \"91a74a01-d2c7-42bd-9ead-8bdc48398e28\") " pod="openshift-dns/dns-default-dbzkb"
Nov 02 01:32:06.615909 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.615829    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:06.615909 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.615893    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f857cdde-184c-436f-a8bd-3de120963268-cert\") pod \"ingress-canary-s275r\" (UID: \"f857cdde-184c-436f-a8bd-3de120963268\") " pod="openshift-ingress-canary/ingress-canary-s275r"
Nov 02 01:32:06.672890 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.672854    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28zhc\" (UniqueName: \"kubernetes.io/projected/91a74a01-d2c7-42bd-9ead-8bdc48398e28-kube-api-access-28zhc\") pod \"dns-default-dbzkb\" (UID: \"91a74a01-d2c7-42bd-9ead-8bdc48398e28\") " pod="openshift-dns/dns-default-dbzkb"
Nov 02 01:32:06.672890 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.672879    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt4vp\" (UniqueName: \"kubernetes.io/projected/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-kube-api-access-jt4vp\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:06.673066 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.672944    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26fmn\" (UniqueName: \"kubernetes.io/projected/f857cdde-184c-436f-a8bd-3de120963268-kube-api-access-26fmn\") pod \"ingress-canary-s275r\" (UID: \"f857cdde-184c-436f-a8bd-3de120963268\") " pod="openshift-ingress-canary/ingress-canary-s275r"
Nov 02 01:32:06.680727 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.680707    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dbzkb"
Nov 02 01:32:06.689836 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.689816    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:06.692251 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:06.692237    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s275r"
Nov 02 01:32:07.063943 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:07.063908    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s275r"]
Nov 02 01:32:07.067630 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:32:07.067604    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf857cdde_184c_436f_a8bd_3de120963268.slice/crio-f65a11b3eb88e9616d6d3b20d732fe322d46788e9eb4fea4e0de2f2c967838c5 WatchSource:0}: Error finding container f65a11b3eb88e9616d6d3b20d732fe322d46788e9eb4fea4e0de2f2c967838c5: Status 404 returned error can't find the container with id f65a11b3eb88e9616d6d3b20d732fe322d46788e9eb4fea4e0de2f2c967838c5
Nov 02 01:32:07.113122 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:07.113096    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dbzkb"]
Nov 02 01:32:07.117558 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:32:07.117527    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a74a01_d2c7_42bd_9ead_8bdc48398e28.slice/crio-b93ed80f94e779453d490ac6563b3bc4fe54645d777edb0179458eb585a87bb1 WatchSource:0}: Error finding container b93ed80f94e779453d490ac6563b3bc4fe54645d777edb0179458eb585a87bb1: Status 404 returned error can't find the container with id b93ed80f94e779453d490ac6563b3bc4fe54645d777edb0179458eb585a87bb1
Nov 02 01:32:07.119387 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:07.117618    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r4njj"]
Nov 02 01:32:07.121093 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:32:07.121067    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fbc2bb9_ce73_41fe_b36f_39c4e3ea4a59.slice/crio-f9ea61f3a326dc5e719210e4d5309b24a1bbcad0df54b4d76cb0258695ad9641 WatchSource:0}: Error finding container f9ea61f3a326dc5e719210e4d5309b24a1bbcad0df54b4d76cb0258695ad9641: Status 404 returned error can't find the container with id f9ea61f3a326dc5e719210e4d5309b24a1bbcad0df54b4d76cb0258695ad9641
Nov 02 01:32:07.495882 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:07.495849    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:32:07.496360 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:07.495894    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:32:07.548169 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:07.548129    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 01:32:07.548327 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:07.548215    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fxpjk\""
Nov 02 01:32:07.548327 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:07.548269    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 01:32:07.548327 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:07.548281    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-m64v5\""
Nov 02 01:32:07.727092 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:07.727057    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s275r" event={"ID":"f857cdde-184c-436f-a8bd-3de120963268","Type":"ContainerStarted","Data":"f65a11b3eb88e9616d6d3b20d732fe322d46788e9eb4fea4e0de2f2c967838c5"}
Nov 02 01:32:07.728479 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:07.728459    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r4njj" event={"ID":"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59","Type":"ContainerStarted","Data":"b784afaac372bb5166f1037789ac187f78eb5c0d0f9638105d5902054f14d767"}
Nov 02 01:32:07.728580 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:07.728502    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r4njj" event={"ID":"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59","Type":"ContainerStarted","Data":"f9ea61f3a326dc5e719210e4d5309b24a1bbcad0df54b4d76cb0258695ad9641"}
Nov 02 01:32:07.729536 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:07.729518    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dbzkb" event={"ID":"91a74a01-d2c7-42bd-9ead-8bdc48398e28","Type":"ContainerStarted","Data":"b93ed80f94e779453d490ac6563b3bc4fe54645d777edb0179458eb585a87bb1"}
Nov 02 01:32:08.734153 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:08.734114    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r4njj" event={"ID":"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59","Type":"ContainerStarted","Data":"db0e94842e9e2ebb78980d865a9cc651f0550b7308242867013b7f3c3caa9fd6"}
Nov 02 01:32:09.740757 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:09.740720    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dbzkb" event={"ID":"91a74a01-d2c7-42bd-9ead-8bdc48398e28","Type":"ContainerStarted","Data":"5f2c50c44511fb57c71992124eb2af9275b724fe259e2a4902f208a419696a47"}
Nov 02 01:32:09.741244 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:09.740792    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dbzkb" event={"ID":"91a74a01-d2c7-42bd-9ead-8bdc48398e28","Type":"ContainerStarted","Data":"9b5e2a24669d56c63ac844db65a8b7f6e217350a0bbf0cc7b5ae43f5b7ba740f"}
Nov 02 01:32:09.741244 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:09.740905    2547 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dbzkb"
Nov 02 01:32:10.106996 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:10.106879    2547 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-dbzkb" podStartSLOduration=1.9377113609999999 podStartE2EDuration="4.106853738s" totalImagesPullingTime="2.169142377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:09.860733611 +0000 UTC m=+51.912049370" watchObservedRunningTime="2026-11-02 01:32:10.106853738 +0000 UTC m=+52.158169500"
Nov 02 01:32:10.107577 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:10.107546    2547 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kpwsv"]
Nov 02 01:32:10.109756 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:10.109732    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kpwsv"
Nov 02 01:32:10.161858 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:10.161834    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 02 01:32:10.161970 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:10.161953    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-xthzw\""
Nov 02 01:32:10.168604 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:10.168550    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kpwsv"]
Nov 02 01:32:10.239166 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:10.239134    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/85999952-cf96-48d1-a608-91c84a7bc17c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-kpwsv\" (UID: \"85999952-cf96-48d1-a608-91c84a7bc17c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kpwsv"
Nov 02 01:32:10.339696 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:10.339653    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/85999952-cf96-48d1-a608-91c84a7bc17c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-kpwsv\" (UID: \"85999952-cf96-48d1-a608-91c84a7bc17c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kpwsv"
Nov 02 01:32:10.342101 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:10.342072    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/85999952-cf96-48d1-a608-91c84a7bc17c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-kpwsv\" (UID: \"85999952-cf96-48d1-a608-91c84a7bc17c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kpwsv"
Nov 02 01:32:10.422330 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:10.422241    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kpwsv"
Nov 02 01:32:10.746123 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:10.746087    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r4njj" event={"ID":"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59","Type":"ContainerStarted","Data":"a7f38736fabd1e9fc92f3a5efc4138f7d0e2b40666c8eb61fa3dbbd668824a88"}
Nov 02 01:32:10.793266 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:10.793216    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kpwsv"]
Nov 02 01:32:10.796635 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:32:10.796604    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85999952_cf96_48d1_a608_91c84a7bc17c.slice/crio-3a1f4347a710d8d6bf041568b564311aa0529b25267be62fb4e42109c6a0afae WatchSource:0}: Error finding container 3a1f4347a710d8d6bf041568b564311aa0529b25267be62fb4e42109c6a0afae: Status 404 returned error can't find the container with id 3a1f4347a710d8d6bf041568b564311aa0529b25267be62fb4e42109c6a0afae
Nov 02 01:32:10.864636 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:10.864572    2547 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-r4njj" podStartSLOduration=1.956331063 podStartE2EDuration="4.864551285s" totalImagesPullingTime="2.908220222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:32:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:10.863500432 +0000 UTC m=+52.914816210" watchObservedRunningTime="2026-11-02 01:32:10.864551285 +0000 UTC m=+52.915867045"
Nov 02 01:32:11.750538 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:11.750500    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s275r" event={"ID":"f857cdde-184c-436f-a8bd-3de120963268","Type":"ContainerStarted","Data":"926824216bcc2ffc7ffe1fda03c47e89a01ff5702d96c24b88a5ef413a98010a"}
Nov 02 01:32:11.751804 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:11.751781    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kpwsv" event={"ID":"85999952-cf96-48d1-a608-91c84a7bc17c","Type":"ContainerStarted","Data":"3a1f4347a710d8d6bf041568b564311aa0529b25267be62fb4e42109c6a0afae"}
Nov 02 01:32:12.756518 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:12.756486    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kpwsv" event={"ID":"85999952-cf96-48d1-a608-91c84a7bc17c","Type":"ContainerStarted","Data":"f92bb372f74aa4b6789a4a82b74163164b5344e34cf5752ce3b2d0b9d34b9e1d"}
Nov 02 01:32:12.756966 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:12.756772    2547 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kpwsv"
Nov 02 01:32:12.761461 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:12.761441    2547 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kpwsv"
Nov 02 01:32:12.869947 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:12.869886    2547 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-s275r" podStartSLOduration=2.70320027 podStartE2EDuration="6.869870306s" totalImagesPullingTime="4.166670036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:11.86694159 +0000 UTC m=+53.918257361" watchObservedRunningTime="2026-11-02 01:32:12.869870306 +0000 UTC m=+54.921186065"
Nov 02 01:32:12.870355 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:12.870325    2547 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kpwsv" podStartSLOduration=1.297197455 podStartE2EDuration="2.870320099s" totalImagesPullingTime="1.573122644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:12.869830141 +0000 UTC m=+54.921145896" watchObservedRunningTime="2026-11-02 01:32:12.870320099 +0000 UTC m=+54.921635856"
Nov 02 01:32:13.723631 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:13.723596    2547 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk"]
Nov 02 01:32:13.725354 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:13.725339    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk"
Nov 02 01:32:13.762943 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:13.762909    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkc2j\" (UniqueName: \"kubernetes.io/projected/792b1392-5f44-47a8-9b86-05ee811046ea-kube-api-access-pkc2j\") pod \"prometheus-operator-9cd89b74c-8zqhk\" (UID: \"792b1392-5f44-47a8-9b86-05ee811046ea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk"
Nov 02 01:32:13.763337 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:13.763036    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/792b1392-5f44-47a8-9b86-05ee811046ea-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8zqhk\" (UID: \"792b1392-5f44-47a8-9b86-05ee811046ea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk"
Nov 02 01:32:13.763337 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:13.763060    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/792b1392-5f44-47a8-9b86-05ee811046ea-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8zqhk\" (UID: \"792b1392-5f44-47a8-9b86-05ee811046ea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk"
Nov 02 01:32:13.763337 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:13.763080    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/792b1392-5f44-47a8-9b86-05ee811046ea-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8zqhk\" (UID: \"792b1392-5f44-47a8-9b86-05ee811046ea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk"
Nov 02 01:32:13.826561 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:13.826522    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 02 01:32:13.826561 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:13.826532    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 02 01:32:13.826767 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:13.826589    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 01:32:13.826767 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:13.826651    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qcsql\""
Nov 02 01:32:13.826767 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:13.826734    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 01:32:13.826956 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:13.826942    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 01:32:13.864201 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:13.864168    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/792b1392-5f44-47a8-9b86-05ee811046ea-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8zqhk\" (UID: \"792b1392-5f44-47a8-9b86-05ee811046ea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk"
Nov 02 01:32:13.864201 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:13.864199    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/792b1392-5f44-47a8-9b86-05ee811046ea-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8zqhk\" (UID: \"792b1392-5f44-47a8-9b86-05ee811046ea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk"
Nov 02 01:32:13.864411 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:13.864244    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/792b1392-5f44-47a8-9b86-05ee811046ea-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8zqhk\" (UID: \"792b1392-5f44-47a8-9b86-05ee811046ea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk"
Nov 02 01:32:13.864411 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:13.864271    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkc2j\" (UniqueName: \"kubernetes.io/projected/792b1392-5f44-47a8-9b86-05ee811046ea-kube-api-access-pkc2j\") pod \"prometheus-operator-9cd89b74c-8zqhk\" (UID: \"792b1392-5f44-47a8-9b86-05ee811046ea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk"
Nov 02 01:32:13.864902 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:13.864875    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/792b1392-5f44-47a8-9b86-05ee811046ea-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8zqhk\" (UID: \"792b1392-5f44-47a8-9b86-05ee811046ea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk"
Nov 02 01:32:13.866786 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:13.866767    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/792b1392-5f44-47a8-9b86-05ee811046ea-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8zqhk\" (UID: \"792b1392-5f44-47a8-9b86-05ee811046ea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk"
Nov 02 01:32:13.866786 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:13.866773    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/792b1392-5f44-47a8-9b86-05ee811046ea-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8zqhk\" (UID: \"792b1392-5f44-47a8-9b86-05ee811046ea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk"
Nov 02 01:32:13.877019 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:13.876998    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk"]
Nov 02 01:32:13.929120 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:13.929087    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkc2j\" (UniqueName: \"kubernetes.io/projected/792b1392-5f44-47a8-9b86-05ee811046ea-kube-api-access-pkc2j\") pod \"prometheus-operator-9cd89b74c-8zqhk\" (UID: \"792b1392-5f44-47a8-9b86-05ee811046ea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk"
Nov 02 01:32:14.035384 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:14.035287    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk"
Nov 02 01:32:14.403786 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:14.403692    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk"]
Nov 02 01:32:14.435874 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:32:14.435842    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792b1392_5f44_47a8_9b86_05ee811046ea.slice/crio-31a3b3106c6b4575f445dafbec4c249d38b1223bef155785bb0ea2faf6ec9f5c WatchSource:0}: Error finding container 31a3b3106c6b4575f445dafbec4c249d38b1223bef155785bb0ea2faf6ec9f5c: Status 404 returned error can't find the container with id 31a3b3106c6b4575f445dafbec4c249d38b1223bef155785bb0ea2faf6ec9f5c
Nov 02 01:32:14.690820 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:14.690737    2547 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:14.766329 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:14.766301    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk" event={"ID":"792b1392-5f44-47a8-9b86-05ee811046ea","Type":"ContainerStarted","Data":"31a3b3106c6b4575f445dafbec4c249d38b1223bef155785bb0ea2faf6ec9f5c"}
Nov 02 01:32:16.779842 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:16.779804    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk" event={"ID":"792b1392-5f44-47a8-9b86-05ee811046ea","Type":"ContainerStarted","Data":"9b7a419bb629407d7d95ae7ae553de0e821f71396f86acf5db9c2be9ebf80dd8"}
Nov 02 01:32:16.780254 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:16.779902    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk" event={"ID":"792b1392-5f44-47a8-9b86-05ee811046ea","Type":"ContainerStarted","Data":"ba4d3145007d42bd9ead4ae672883c857e6a27a883cbb7718ffbd9c1f2e9b3d7"}
Nov 02 01:32:16.895856 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:16.895800    2547 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk" podStartSLOduration=2.100400565 podStartE2EDuration="3.895786668s" totalImagesPullingTime="1.795386103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:16.895652466 +0000 UTC m=+58.946968242" watchObservedRunningTime="2026-11-02 01:32:16.895786668 +0000 UTC m=+58.947102426"
Nov 02 01:32:19.748338 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:19.748306    2547 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dbzkb"
Nov 02 01:32:20.247958 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.247925    2547 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"]
Nov 02 01:32:20.251481 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.251458    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"
Nov 02 01:32:20.303667 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.303634    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 02 01:32:20.305999 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.305980    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f86263ab-30e1-40bf-a671-f82f44423125-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-4snk7\" (UID: \"f86263ab-30e1-40bf-a671-f82f44423125\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"
Nov 02 01:32:20.306058 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.306041    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zkz5\" (UniqueName: \"kubernetes.io/projected/f86263ab-30e1-40bf-a671-f82f44423125-kube-api-access-4zkz5\") pod \"openshift-state-metrics-7bd7489957-4snk7\" (UID: \"f86263ab-30e1-40bf-a671-f82f44423125\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"
Nov 02 01:32:20.306156 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.306134    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f86263ab-30e1-40bf-a671-f82f44423125-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-4snk7\" (UID: \"f86263ab-30e1-40bf-a671-f82f44423125\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"
Nov 02 01:32:20.306213 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.306198    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f86263ab-30e1-40bf-a671-f82f44423125-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-4snk7\" (UID: \"f86263ab-30e1-40bf-a671-f82f44423125\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"
Nov 02 01:32:20.351987 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.351950    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 02 01:32:20.351987 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.351990    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8tdqx\""
Nov 02 01:32:20.353991 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.353974    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"]
Nov 02 01:32:20.367100 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.367072    2547 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"]
Nov 02 01:32:20.369800 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.369780    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:20.402667 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.402639    2547 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rrtpw"]
Nov 02 01:32:20.405548 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.405529    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.407096 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.407073    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f70370d-3793-4e25-90e8-fc252f5a7596-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:20.407164 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.407105    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zkz5\" (UniqueName: \"kubernetes.io/projected/f86263ab-30e1-40bf-a671-f82f44423125-kube-api-access-4zkz5\") pod \"openshift-state-metrics-7bd7489957-4snk7\" (UID: \"f86263ab-30e1-40bf-a671-f82f44423125\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"
Nov 02 01:32:20.407209 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.407181    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f86263ab-30e1-40bf-a671-f82f44423125-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-4snk7\" (UID: \"f86263ab-30e1-40bf-a671-f82f44423125\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"
Nov 02 01:32:20.407276 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.407243    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f70370d-3793-4e25-90e8-fc252f5a7596-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:20.407326 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.407283    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f86263ab-30e1-40bf-a671-f82f44423125-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-4snk7\" (UID: \"f86263ab-30e1-40bf-a671-f82f44423125\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"
Nov 02 01:32:20.407386 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.407354    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f70370d-3793-4e25-90e8-fc252f5a7596-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:20.407436 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.407391    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5f70370d-3793-4e25-90e8-fc252f5a7596-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:20.407486 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.407469    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5f70370d-3793-4e25-90e8-fc252f5a7596-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:20.407567 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.407546    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f86263ab-30e1-40bf-a671-f82f44423125-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-4snk7\" (UID: \"f86263ab-30e1-40bf-a671-f82f44423125\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"
Nov 02 01:32:20.407628 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.407585    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hl42\" (UniqueName: \"kubernetes.io/projected/5f70370d-3793-4e25-90e8-fc252f5a7596-kube-api-access-8hl42\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:20.408479 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.408457    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f86263ab-30e1-40bf-a671-f82f44423125-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-4snk7\" (UID: \"f86263ab-30e1-40bf-a671-f82f44423125\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"
Nov 02 01:32:20.409683 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.409662    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f86263ab-30e1-40bf-a671-f82f44423125-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-4snk7\" (UID: \"f86263ab-30e1-40bf-a671-f82f44423125\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"
Nov 02 01:32:20.409777 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.409758    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f86263ab-30e1-40bf-a671-f82f44423125-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-4snk7\" (UID: \"f86263ab-30e1-40bf-a671-f82f44423125\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"
Nov 02 01:32:20.424941 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.424922    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 02 01:32:20.427580 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.427559    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 02 01:32:20.427761 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.427743    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-thfsj\""
Nov 02 01:32:20.428106 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.428093    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 02 01:32:20.435666 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.435637    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"]
Nov 02 01:32:20.468094 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.468055    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zkz5\" (UniqueName: \"kubernetes.io/projected/f86263ab-30e1-40bf-a671-f82f44423125-kube-api-access-4zkz5\") pod \"openshift-state-metrics-7bd7489957-4snk7\" (UID: \"f86263ab-30e1-40bf-a671-f82f44423125\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"
Nov 02 01:32:20.508524 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.508446    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.508524 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.508481    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f70370d-3793-4e25-90e8-fc252f5a7596-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:20.508524 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.508498    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-metrics-client-ca\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.508524 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.508520    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-textfile\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.508764 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.508572    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-root\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.508764 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.508635    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5f70370d-3793-4e25-90e8-fc252f5a7596-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:20.508764 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.508655    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hl42\" (UniqueName: \"kubernetes.io/projected/5f70370d-3793-4e25-90e8-fc252f5a7596-kube-api-access-8hl42\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:20.508764 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.508695    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-wtmp\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.508764 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.508740    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-sys\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.508764 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.508759    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.509042 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.508783    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f70370d-3793-4e25-90e8-fc252f5a7596-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:20.509042 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.508834    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f70370d-3793-4e25-90e8-fc252f5a7596-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:20.509042 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.508889    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5f70370d-3793-4e25-90e8-fc252f5a7596-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:20.509042 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.509007    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77lwg\" (UniqueName: \"kubernetes.io/projected/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-kube-api-access-77lwg\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.509270 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.509010    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5f70370d-3793-4e25-90e8-fc252f5a7596-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:20.509270 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.509079    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-tls\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.509270 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.509262    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f70370d-3793-4e25-90e8-fc252f5a7596-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:20.509720 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.509697    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5f70370d-3793-4e25-90e8-fc252f5a7596-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:20.511398 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.511381    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f70370d-3793-4e25-90e8-fc252f5a7596-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:20.511453 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.511433    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f70370d-3793-4e25-90e8-fc252f5a7596-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:20.525130 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.525108    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 01:32:20.525395 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.525381    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6vdkx\""
Nov 02 01:32:20.525486 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.525472    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 01:32:20.528021 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.528008    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 01:32:20.564619 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.564590    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"
Nov 02 01:32:20.569984 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.569963    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hl42\" (UniqueName: \"kubernetes.io/projected/5f70370d-3793-4e25-90e8-fc252f5a7596-kube-api-access-8hl42\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:20.609472 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.609439    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-tls\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.609647 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.609507    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.609647 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.609534    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-metrics-client-ca\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.609647 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.609567    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-textfile\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.609647 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.609588    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-root\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.609647 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.609618    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-wtmp\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.609889 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.609661    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-sys\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.609889 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.609688    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.609889 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.609733    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77lwg\" (UniqueName: \"kubernetes.io/projected/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-kube-api-access-77lwg\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.610035 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.609901    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-wtmp\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.610445 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.610293    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-sys\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.610445 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.610365    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-root\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.610445 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.610481    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-textfile\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.610875 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.610851    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-metrics-client-ca\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.611448 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.611345    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.612280 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.612238    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.616418 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.614756    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-tls\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.670656 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.670627    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77lwg\" (UniqueName: \"kubernetes.io/projected/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-kube-api-access-77lwg\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.684293 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.684260    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:20.729078 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.729040    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:20.743049 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:32:20.743020    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bbd7104_ba74_4c31_8cfb_3f307da4b3f4.slice/crio-746e119d52d73097b35d0921c93fddceeafd18cdc55e96cb368f9b9a5c54d5be WatchSource:0}: Error finding container 746e119d52d73097b35d0921c93fddceeafd18cdc55e96cb368f9b9a5c54d5be: Status 404 returned error can't find the container with id 746e119d52d73097b35d0921c93fddceeafd18cdc55e96cb368f9b9a5c54d5be
Nov 02 01:32:20.793664 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.793586    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rrtpw" event={"ID":"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4","Type":"ContainerStarted","Data":"746e119d52d73097b35d0921c93fddceeafd18cdc55e96cb368f9b9a5c54d5be"}
Nov 02 01:32:20.983855 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:20.983824    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"]
Nov 02 01:32:20.986407 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:32:20.986378    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf86263ab_30e1_40bf_a671_f82f44423125.slice/crio-9ea0457b904929e89ca2f6b5468317750d056f07850e062ed00d655a5c8ad218 WatchSource:0}: Error finding container 9ea0457b904929e89ca2f6b5468317750d056f07850e062ed00d655a5c8ad218: Status 404 returned error can't find the container with id 9ea0457b904929e89ca2f6b5468317750d056f07850e062ed00d655a5c8ad218
Nov 02 01:32:21.097785 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:21.097766    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"]
Nov 02 01:32:21.099754 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:32:21.099725    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f70370d_3793_4e25_90e8_fc252f5a7596.slice/crio-0db0a4440d26849b6911b476f8ddda7e31933f0ea3721836115dae07380692b7 WatchSource:0}: Error finding container 0db0a4440d26849b6911b476f8ddda7e31933f0ea3721836115dae07380692b7: Status 404 returned error can't find the container with id 0db0a4440d26849b6911b476f8ddda7e31933f0ea3721836115dae07380692b7
Nov 02 01:32:21.799178 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:21.799139    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7" event={"ID":"f86263ab-30e1-40bf-a671-f82f44423125","Type":"ContainerStarted","Data":"15c550059c5c7f85df7b1963742ba9218c32de91420d4041eb5bb26185c93143"}
Nov 02 01:32:21.799660 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:21.799214    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7" event={"ID":"f86263ab-30e1-40bf-a671-f82f44423125","Type":"ContainerStarted","Data":"2fc01cc064703cef15c7bbb9165eb1285e808191e1795da26c2e7b804bd6d37a"}
Nov 02 01:32:21.799660 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:21.799365    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7" event={"ID":"f86263ab-30e1-40bf-a671-f82f44423125","Type":"ContainerStarted","Data":"9ea0457b904929e89ca2f6b5468317750d056f07850e062ed00d655a5c8ad218"}
Nov 02 01:32:21.801121 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:21.801095    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r" event={"ID":"5f70370d-3793-4e25-90e8-fc252f5a7596","Type":"ContainerStarted","Data":"0db0a4440d26849b6911b476f8ddda7e31933f0ea3721836115dae07380692b7"}
Nov 02 01:32:22.439498 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.439428    2547 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 01:32:22.448420 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.448391    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.501429 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.501373    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 02 01:32:22.526547 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.526502    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.526685 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.526554    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpswh\" (UniqueName: \"kubernetes.io/projected/0e0fdbea-232a-4a5e-abe4-3869da804c72-kube-api-access-gpswh\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.526750 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.526723    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.526791 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.526773    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0e0fdbea-232a-4a5e-abe4-3869da804c72-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.526849 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.526832    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e0fdbea-232a-4a5e-abe4-3869da804c72-config-out\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.526957 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.526940    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-config-volume\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.527006 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.526989    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.527061 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.527046    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e0fdbea-232a-4a5e-abe4-3869da804c72-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.527093 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.527074    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e0fdbea-232a-4a5e-abe4-3869da804c72-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.527134 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.527099    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.527173 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.527137    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-web-config\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.527173 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.527152    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e0fdbea-232a-4a5e-abe4-3869da804c72-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.527288 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.527193    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.551074 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.550489    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 02 01:32:22.551074 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.550686    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 02 01:32:22.551074 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.550728    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 02 01:32:22.551074 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.550782    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 02 01:32:22.551074 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.550905    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 02 01:32:22.551074 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.551019    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 02 01:32:22.551631 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.551610    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-grbp4\""
Nov 02 01:32:22.551748 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.551655    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 02 01:32:22.600819 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.600742    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 02 01:32:22.603215 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.603189    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 01:32:22.628402 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.627617    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0e0fdbea-232a-4a5e-abe4-3869da804c72-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.628402 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.627662    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e0fdbea-232a-4a5e-abe4-3869da804c72-config-out\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.628402 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.627696    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-config-volume\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.628402 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.627742    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.628402 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.627776    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e0fdbea-232a-4a5e-abe4-3869da804c72-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.628402 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.627799    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e0fdbea-232a-4a5e-abe4-3869da804c72-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.628402 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.627813    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.628402 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.627834    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-web-config\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.628402 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.627850    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e0fdbea-232a-4a5e-abe4-3869da804c72-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.628402 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.627881    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.628402 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.627911    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.628402 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.627935    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpswh\" (UniqueName: \"kubernetes.io/projected/0e0fdbea-232a-4a5e-abe4-3869da804c72-kube-api-access-gpswh\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.628402 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.627999    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.628402 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.628070    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0e0fdbea-232a-4a5e-abe4-3869da804c72-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.629369 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.628484    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e0fdbea-232a-4a5e-abe4-3869da804c72-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.629369 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.628718    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e0fdbea-232a-4a5e-abe4-3869da804c72-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.631015 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.630973    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.631482 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.631446    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e0fdbea-232a-4a5e-abe4-3869da804c72-config-out\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.631586 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.631528    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.633050 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.632954    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e0fdbea-232a-4a5e-abe4-3869da804c72-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.634615 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.634568    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.636677 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.636654    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.637001 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.636975    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-web-config\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.637167 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.637124    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.637485 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.637464    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-config-volume\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.690392 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.690309    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpswh\" (UniqueName: \"kubernetes.io/projected/0e0fdbea-232a-4a5e-abe4-3869da804c72-kube-api-access-gpswh\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.777261 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.777211    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:22.807106 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.807071    2547 generic.go:356] "Generic (PLEG): container finished" podID="9bbd7104-ba74-4c31-8cfb-3f307da4b3f4" containerID="2bef5272c303e612a2d2e915b53534d9872ae79ec6593e02f8ecddd7b6caf3b9" exitCode=0
Nov 02 01:32:22.807537 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.807128    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rrtpw" event={"ID":"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4","Type":"ContainerDied","Data":"2bef5272c303e612a2d2e915b53534d9872ae79ec6593e02f8ecddd7b6caf3b9"}
Nov 02 01:32:22.884264 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.884238    2547 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-b84687bcd-x24hf"]
Nov 02 01:32:22.888034 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.888017    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:22.931583 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.931465    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-tls\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:22.931702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.931605    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-grpc-tls\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:22.931702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.931666    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:22.931702 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.931697    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:22.931814 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.931720    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:22.931814 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.931746    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c333281d-bd01-4c24-bb15-fa3ee2583fb9-tmp\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:22.931883 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.931846    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c333281d-bd01-4c24-bb15-fa3ee2583fb9-metrics-client-ca\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:22.931958 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.931939    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:22.932001 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.931973    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8hv9\" (UniqueName: \"kubernetes.io/projected/c333281d-bd01-4c24-bb15-fa3ee2583fb9-kube-api-access-m8hv9\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:22.945018 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.944954    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 02 01:32:22.945146 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.945033    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 02 01:32:22.945252 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.945168    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-swgv6\""
Nov 02 01:32:22.945252 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.945212    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3rm6nv61eub59\""
Nov 02 01:32:22.945367 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.945270    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 02 01:32:22.945367 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.945315    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 02 01:32:22.945455 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.945373    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 02 01:32:22.954755 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:22.954725    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b84687bcd-x24hf"]
Nov 02 01:32:23.033344 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.033313    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-tls\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:23.033511 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.033358    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-grpc-tls\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:23.033511 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.033392    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:23.033511 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.033421    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:23.033670 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.033584    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:23.033670 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.033629    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c333281d-bd01-4c24-bb15-fa3ee2583fb9-tmp\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:23.033774 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.033681    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c333281d-bd01-4c24-bb15-fa3ee2583fb9-metrics-client-ca\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:23.033774 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.033727    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:23.033774 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.033754    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8hv9\" (UniqueName: \"kubernetes.io/projected/c333281d-bd01-4c24-bb15-fa3ee2583fb9-kube-api-access-m8hv9\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:23.033964 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.033939    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c333281d-bd01-4c24-bb15-fa3ee2583fb9-tmp\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:23.034697 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.034642    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c333281d-bd01-4c24-bb15-fa3ee2583fb9-metrics-client-ca\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:23.036777 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.036436    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:23.036777 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.036450    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-tls\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:23.037242 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.037203    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:23.037317 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.036826    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:23.037615 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.037591    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:23.037663 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.037650    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-grpc-tls\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:23.094174 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.094149    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8hv9\" (UniqueName: \"kubernetes.io/projected/c333281d-bd01-4c24-bb15-fa3ee2583fb9-kube-api-access-m8hv9\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:23.216390 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.216346    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 01:32:23.217654 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.217626    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:23.218976 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:32:23.218948    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e0fdbea_232a_4a5e_abe4_3869da804c72.slice/crio-b178ceb687ed740a6de9b4f920ff574b54f851f0634ea61f239e6902bce2af03 WatchSource:0}: Error finding container b178ceb687ed740a6de9b4f920ff574b54f851f0634ea61f239e6902bce2af03: Status 404 returned error can't find the container with id b178ceb687ed740a6de9b4f920ff574b54f851f0634ea61f239e6902bce2af03
Nov 02 01:32:23.437202 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.437172    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret\") pod \"global-pull-secret-syncer-8vhjc\" (UID: \"b88b4dfb-1523-4701-9388-781cedab823b\") " pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:32:23.437202 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.437203    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs\") pod \"network-metrics-daemon-gws4h\" (UID: \"841f3428-10c6-4079-a256-cf6e5cd21806\") " pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:32:23.488654 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.488589    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 01:32:23.488775 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.488666    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 01:32:23.500183 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.500156    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs\") pod \"network-metrics-daemon-gws4h\" (UID: \"841f3428-10c6-4079-a256-cf6e5cd21806\") " pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:32:23.500335 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.500318    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret\") pod \"global-pull-secret-syncer-8vhjc\" (UID: \"b88b4dfb-1523-4701-9388-781cedab823b\") " pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:32:23.537621 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.537592    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9sj2\" (UniqueName: \"kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2\") pod \"network-check-target-gzq27\" (UID: \"6a2f0fa5-cbce-4695-aa38-e8bbeb720280\") " pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:32:23.588803 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.588769    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 01:32:23.598451 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.598421    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b84687bcd-x24hf"]
Nov 02 01:32:23.601170 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:32:23.601146    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc333281d_bd01_4c24_bb15_fa3ee2583fb9.slice/crio-b6d333f27e662dda79ef0693c4220289ca475279c9948cae04af48d4ce3e68ac WatchSource:0}: Error finding container b6d333f27e662dda79ef0693c4220289ca475279c9948cae04af48d4ce3e68ac: Status 404 returned error can't find the container with id b6d333f27e662dda79ef0693c4220289ca475279c9948cae04af48d4ce3e68ac
Nov 02 01:32:23.624939 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.624910    2547 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-lltpm"]
Nov 02 01:32:23.629110 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.629096    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-lltpm"
Nov 02 01:32:23.649313 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.649281    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 01:32:23.663659 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.663630    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9sj2\" (UniqueName: \"kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2\") pod \"network-check-target-gzq27\" (UID: \"6a2f0fa5-cbce-4695-aa38-e8bbeb720280\") " pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:32:23.681313 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.681285    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-bf747\""
Nov 02 01:32:23.681313 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.681286    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 02 01:32:23.681464 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.681286    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 02 01:32:23.687474 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.687450    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-lltpm"]
Nov 02 01:32:23.738214 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.738182    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5dbcf879-26f8-4f7b-a071-eafa688b1c05-tmp\") pod \"downloads-6dd789c8b7-lltpm\" (UID: \"5dbcf879-26f8-4f7b-a071-eafa688b1c05\") " pod="openshift-console/downloads-6dd789c8b7-lltpm"
Nov 02 01:32:23.738433 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.738279    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kjnt\" (UniqueName: \"kubernetes.io/projected/5dbcf879-26f8-4f7b-a071-eafa688b1c05-kube-api-access-7kjnt\") pod \"downloads-6dd789c8b7-lltpm\" (UID: \"5dbcf879-26f8-4f7b-a071-eafa688b1c05\") " pod="openshift-console/downloads-6dd789c8b7-lltpm"
Nov 02 01:32:23.759433 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.759351    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fxpjk\""
Nov 02 01:32:23.762379 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.762359    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-m64v5\""
Nov 02 01:32:23.768406 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.768380    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:32:23.772396 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.772381    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:32:23.813809 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.813702    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf" event={"ID":"c333281d-bd01-4c24-bb15-fa3ee2583fb9","Type":"ContainerStarted","Data":"b6d333f27e662dda79ef0693c4220289ca475279c9948cae04af48d4ce3e68ac"}
Nov 02 01:32:23.817507 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.817477    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rrtpw" event={"ID":"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4","Type":"ContainerStarted","Data":"96d2926900183b8728836afb6c975ff79285f84dc46874b5170d45253316c0bd"}
Nov 02 01:32:23.817666 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.817525    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rrtpw" event={"ID":"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4","Type":"ContainerStarted","Data":"4f9ffa022fe613b64fa05942b104ab31e25cf9e25ade7e3f90301697fdfde21b"}
Nov 02 01:32:23.819662 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.819638    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7" event={"ID":"f86263ab-30e1-40bf-a671-f82f44423125","Type":"ContainerStarted","Data":"4fe8a277cc37100492882f2187f8683e67d4b8d2fceaccbe49f09055d1863482"}
Nov 02 01:32:23.833801 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.828114    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r" event={"ID":"5f70370d-3793-4e25-90e8-fc252f5a7596","Type":"ContainerStarted","Data":"5c5efd266418c62f5edbfaa49479dcd9bedbba69c73bde0c458348621ac0f247"}
Nov 02 01:32:23.833801 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.828179    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r" event={"ID":"5f70370d-3793-4e25-90e8-fc252f5a7596","Type":"ContainerStarted","Data":"f2cca2dfd0f9de0ac8455f19c59f91dbeee5854ea2015970327423d26fd011c4"}
Nov 02 01:32:23.833801 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.828212    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r" event={"ID":"5f70370d-3793-4e25-90e8-fc252f5a7596","Type":"ContainerStarted","Data":"36f5f97021dbf0fafad0478a893e19db088c7c21d1b4f60e83517f10ee906ba5"}
Nov 02 01:32:23.833801 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.833545    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e0fdbea-232a-4a5e-abe4-3869da804c72","Type":"ContainerStarted","Data":"b178ceb687ed740a6de9b4f920ff574b54f851f0634ea61f239e6902bce2af03"}
Nov 02 01:32:23.839244 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.839183    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kjnt\" (UniqueName: \"kubernetes.io/projected/5dbcf879-26f8-4f7b-a071-eafa688b1c05-kube-api-access-7kjnt\") pod \"downloads-6dd789c8b7-lltpm\" (UID: \"5dbcf879-26f8-4f7b-a071-eafa688b1c05\") " pod="openshift-console/downloads-6dd789c8b7-lltpm"
Nov 02 01:32:23.839356 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.839335    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5dbcf879-26f8-4f7b-a071-eafa688b1c05-tmp\") pod \"downloads-6dd789c8b7-lltpm\" (UID: \"5dbcf879-26f8-4f7b-a071-eafa688b1c05\") " pod="openshift-console/downloads-6dd789c8b7-lltpm"
Nov 02 01:32:23.840956 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.840934    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5dbcf879-26f8-4f7b-a071-eafa688b1c05-tmp\") pod \"downloads-6dd789c8b7-lltpm\" (UID: \"5dbcf879-26f8-4f7b-a071-eafa688b1c05\") " pod="openshift-console/downloads-6dd789c8b7-lltpm"
Nov 02 01:32:23.900360 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.900325    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kjnt\" (UniqueName: \"kubernetes.io/projected/5dbcf879-26f8-4f7b-a071-eafa688b1c05-kube-api-access-7kjnt\") pod \"downloads-6dd789c8b7-lltpm\" (UID: \"5dbcf879-26f8-4f7b-a071-eafa688b1c05\") " pod="openshift-console/downloads-6dd789c8b7-lltpm"
Nov 02 01:32:23.936655 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.936583    2547 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rrtpw" podStartSLOduration=2.434864277 podStartE2EDuration="3.93656559s" totalImagesPullingTime="1.41094313s" totalInitContainerRuntime="90.758183ms" isStatefulPod=true podCreationTimestamp="2026-11-02 01:32:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:23.936024234 +0000 UTC m=+65.987340006" watchObservedRunningTime="2026-11-02 01:32:23.93656559 +0000 UTC m=+65.987881349"
Nov 02 01:32:23.960200 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.956391    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-42sf2\""
Nov 02 01:32:23.961183 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.961158    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-lltpm"
Nov 02 01:32:23.965144 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:23.965123    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:32:24.054111 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:24.054058    2547 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7" podStartSLOduration=2.321316276 podStartE2EDuration="4.054044407s" totalImagesPullingTime="1.732728131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:24.052832302 +0000 UTC m=+66.104148060" watchObservedRunningTime="2026-11-02 01:32:24.054044407 +0000 UTC m=+66.105360165"
Nov 02 01:32:24.148970 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:24.148944    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gws4h"]
Nov 02 01:32:24.151480 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:24.151456    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8vhjc"]
Nov 02 01:32:24.151689 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:32:24.151669    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841f3428_10c6_4079_a256_cf6e5cd21806.slice/crio-069fc7fc54846606f1108d55c7f501f0612a442463133eb380c50e044f2b5eef WatchSource:0}: Error finding container 069fc7fc54846606f1108d55c7f501f0612a442463133eb380c50e044f2b5eef: Status 404 returned error can't find the container with id 069fc7fc54846606f1108d55c7f501f0612a442463133eb380c50e044f2b5eef
Nov 02 01:32:24.153762 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:32:24.153741    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88b4dfb_1523_4701_9388_781cedab823b.slice/crio-b70d62c7a50f16a630ac3255032d18f9af3d701854c30ba24fddf6128ca24204 WatchSource:0}: Error finding container b70d62c7a50f16a630ac3255032d18f9af3d701854c30ba24fddf6128ca24204: Status 404 returned error can't find the container with id b70d62c7a50f16a630ac3255032d18f9af3d701854c30ba24fddf6128ca24204
Nov 02 01:32:24.171466 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:24.171419    2547 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r" podStartSLOduration=2.218204938 podStartE2EDuration="4.171404362s" totalImagesPullingTime="1.953199424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:24.17045025 +0000 UTC m=+66.221766023" watchObservedRunningTime="2026-11-02 01:32:24.171404362 +0000 UTC m=+66.222720353"
Nov 02 01:32:24.342486 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:24.342395    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gzq27"]
Nov 02 01:32:24.345475 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:32:24.345451    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a2f0fa5_cbce_4695_aa38_e8bbeb720280.slice/crio-212349606df6c854d29a533d0079cf270f94912312bbddc935e656a210c5b596 WatchSource:0}: Error finding container 212349606df6c854d29a533d0079cf270f94912312bbddc935e656a210c5b596: Status 404 returned error can't find the container with id 212349606df6c854d29a533d0079cf270f94912312bbddc935e656a210c5b596
Nov 02 01:32:24.348716 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:24.348697    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-lltpm"]
Nov 02 01:32:24.351421 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:32:24.351395    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dbcf879_26f8_4f7b_a071_eafa688b1c05.slice/crio-7dba8084753fc3d3c7fb38ca22890cf10dd689e50a76867c80541fd8ee48baf8 WatchSource:0}: Error finding container 7dba8084753fc3d3c7fb38ca22890cf10dd689e50a76867c80541fd8ee48baf8: Status 404 returned error can't find the container with id 7dba8084753fc3d3c7fb38ca22890cf10dd689e50a76867c80541fd8ee48baf8
Nov 02 01:32:24.839041 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:24.839009    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-lltpm" event={"ID":"5dbcf879-26f8-4f7b-a071-eafa688b1c05","Type":"ContainerStarted","Data":"7dba8084753fc3d3c7fb38ca22890cf10dd689e50a76867c80541fd8ee48baf8"}
Nov 02 01:32:24.840670 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:24.840640    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8vhjc" event={"ID":"b88b4dfb-1523-4701-9388-781cedab823b","Type":"ContainerStarted","Data":"b70d62c7a50f16a630ac3255032d18f9af3d701854c30ba24fddf6128ca24204"}
Nov 02 01:32:24.842077 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:24.842027    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gws4h" event={"ID":"841f3428-10c6-4079-a256-cf6e5cd21806","Type":"ContainerStarted","Data":"069fc7fc54846606f1108d55c7f501f0612a442463133eb380c50e044f2b5eef"}
Nov 02 01:32:24.843518 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:24.843493    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gzq27" event={"ID":"6a2f0fa5-cbce-4695-aa38-e8bbeb720280","Type":"ContainerStarted","Data":"212349606df6c854d29a533d0079cf270f94912312bbddc935e656a210c5b596"}
Nov 02 01:32:24.982206 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:24.982176    2547 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-rc9qm"]
Nov 02 01:32:24.985511 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:24.985436    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rc9qm"
Nov 02 01:32:25.040719 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.040686    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8kxtm\""
Nov 02 01:32:25.040930 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.040872    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 02 01:32:25.042215 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.042147    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-rc9qm"]
Nov 02 01:32:25.052187 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.052126    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e9dd01bc-7209-4fe5-ac79-8efaa6c14232-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rc9qm\" (UID: \"e9dd01bc-7209-4fe5-ac79-8efaa6c14232\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rc9qm"
Nov 02 01:32:25.052522 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.052191    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx2wj\" (UniqueName: \"kubernetes.io/projected/e9dd01bc-7209-4fe5-ac79-8efaa6c14232-kube-api-access-qx2wj\") pod \"monitoring-plugin-77f4f76594-rc9qm\" (UID: \"e9dd01bc-7209-4fe5-ac79-8efaa6c14232\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rc9qm"
Nov 02 01:32:25.147414 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.145996    2547 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-79ffdc6779-dp2ff"]
Nov 02 01:32:25.149530 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.149506    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:25.153575 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.153548    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx2wj\" (UniqueName: \"kubernetes.io/projected/e9dd01bc-7209-4fe5-ac79-8efaa6c14232-kube-api-access-qx2wj\") pod \"monitoring-plugin-77f4f76594-rc9qm\" (UID: \"e9dd01bc-7209-4fe5-ac79-8efaa6c14232\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rc9qm"
Nov 02 01:32:25.153676 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.153626    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e9dd01bc-7209-4fe5-ac79-8efaa6c14232-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rc9qm\" (UID: \"e9dd01bc-7209-4fe5-ac79-8efaa6c14232\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rc9qm"
Nov 02 01:32:25.156676 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.156624    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e9dd01bc-7209-4fe5-ac79-8efaa6c14232-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rc9qm\" (UID: \"e9dd01bc-7209-4fe5-ac79-8efaa6c14232\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rc9qm"
Nov 02 01:32:25.201992 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.201960    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 02 01:32:25.202162 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.202073    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 02 01:32:25.204120 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.203725    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-lv26r\""
Nov 02 01:32:25.204120 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.203825    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 02 01:32:25.204120 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.204074    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-es4kmf43m5t2c\""
Nov 02 01:32:25.204398 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.204196    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 02 01:32:25.209426 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.209398    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-79ffdc6779-dp2ff"]
Nov 02 01:32:25.215654 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.215631    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx2wj\" (UniqueName: \"kubernetes.io/projected/e9dd01bc-7209-4fe5-ac79-8efaa6c14232-kube-api-access-qx2wj\") pod \"monitoring-plugin-77f4f76594-rc9qm\" (UID: \"e9dd01bc-7209-4fe5-ac79-8efaa6c14232\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rc9qm"
Nov 02 01:32:25.254755 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.254718    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/544ee005-9680-4d16-ae14-a2633763bf10-secret-metrics-server-tls\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:25.254962 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.254771    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/544ee005-9680-4d16-ae14-a2633763bf10-client-ca-bundle\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:25.254962 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.254884    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/544ee005-9680-4d16-ae14-a2633763bf10-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:25.255049 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.254999    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/544ee005-9680-4d16-ae14-a2633763bf10-secret-metrics-server-client-certs\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:25.255049 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.255031    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/544ee005-9680-4d16-ae14-a2633763bf10-audit-log\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:25.255143 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.255060    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdrcw\" (UniqueName: \"kubernetes.io/projected/544ee005-9680-4d16-ae14-a2633763bf10-kube-api-access-rdrcw\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:25.255143 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.255109    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/544ee005-9680-4d16-ae14-a2633763bf10-metrics-server-audit-profiles\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:25.355923 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.355882    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/544ee005-9680-4d16-ae14-a2633763bf10-secret-metrics-server-client-certs\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:25.356083 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.355942    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/544ee005-9680-4d16-ae14-a2633763bf10-audit-log\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:25.356083 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.355970    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdrcw\" (UniqueName: \"kubernetes.io/projected/544ee005-9680-4d16-ae14-a2633763bf10-kube-api-access-rdrcw\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:25.356083 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.356016    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/544ee005-9680-4d16-ae14-a2633763bf10-metrics-server-audit-profiles\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:25.356083 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.356057    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/544ee005-9680-4d16-ae14-a2633763bf10-secret-metrics-server-tls\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:25.356269 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.356093    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/544ee005-9680-4d16-ae14-a2633763bf10-client-ca-bundle\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:25.356269 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.356132    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/544ee005-9680-4d16-ae14-a2633763bf10-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:25.356834 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.356811    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/544ee005-9680-4d16-ae14-a2633763bf10-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:25.357802 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.357482    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/544ee005-9680-4d16-ae14-a2633763bf10-audit-log\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:25.358028 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.358007    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/544ee005-9680-4d16-ae14-a2633763bf10-metrics-server-audit-profiles\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:25.358691 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.358669    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rc9qm"
Nov 02 01:32:25.360439 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.360387    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/544ee005-9680-4d16-ae14-a2633763bf10-client-ca-bundle\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:25.360848 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.360814    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/544ee005-9680-4d16-ae14-a2633763bf10-secret-metrics-server-tls\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:25.361100 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.361078    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/544ee005-9680-4d16-ae14-a2633763bf10-secret-metrics-server-client-certs\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:25.424660 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.424555    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdrcw\" (UniqueName: \"kubernetes.io/projected/544ee005-9680-4d16-ae14-a2633763bf10-kube-api-access-rdrcw\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:25.467251 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.466950    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:25.754615 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.754588    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-rc9qm"]
Nov 02 01:32:25.758119 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:32:25.758084    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9dd01bc_7209_4fe5_ac79_8efaa6c14232.slice/crio-8f9a4968a094313736ea605c9a4f7a4883f0786b353a37a3e88d8c9fa38ad501 WatchSource:0}: Error finding container 8f9a4968a094313736ea605c9a4f7a4883f0786b353a37a3e88d8c9fa38ad501: Status 404 returned error can't find the container with id 8f9a4968a094313736ea605c9a4f7a4883f0786b353a37a3e88d8c9fa38ad501
Nov 02 01:32:25.852339 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.851592    2547 generic.go:356] "Generic (PLEG): container finished" podID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerID="3ee78722ab1691cad1750ad559dc90bcb26c8b3c3120835a012d2d0e70cbeeba" exitCode=0
Nov 02 01:32:25.852339 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.851675    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e0fdbea-232a-4a5e-abe4-3869da804c72","Type":"ContainerDied","Data":"3ee78722ab1691cad1750ad559dc90bcb26c8b3c3120835a012d2d0e70cbeeba"}
Nov 02 01:32:25.855441 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.855418    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rc9qm" event={"ID":"e9dd01bc-7209-4fe5-ac79-8efaa6c14232","Type":"ContainerStarted","Data":"8f9a4968a094313736ea605c9a4f7a4883f0786b353a37a3e88d8c9fa38ad501"}
Nov 02 01:32:25.866902 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:25.866827    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-79ffdc6779-dp2ff"]
Nov 02 01:32:25.872300 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:32:25.872271    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod544ee005_9680_4d16_ae14_a2633763bf10.slice/crio-7e005dea061ad10babcd2a270d689a9cb0c2b5c78f224d8c1328fd08fda7c2e0 WatchSource:0}: Error finding container 7e005dea061ad10babcd2a270d689a9cb0c2b5c78f224d8c1328fd08fda7c2e0: Status 404 returned error can't find the container with id 7e005dea061ad10babcd2a270d689a9cb0c2b5c78f224d8c1328fd08fda7c2e0
Nov 02 01:32:26.865612 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:26.865511    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff" event={"ID":"544ee005-9680-4d16-ae14-a2633763bf10","Type":"ContainerStarted","Data":"7e005dea061ad10babcd2a270d689a9cb0c2b5c78f224d8c1328fd08fda7c2e0"}
Nov 02 01:32:26.872183 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:26.872152    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gws4h" event={"ID":"841f3428-10c6-4079-a256-cf6e5cd21806","Type":"ContainerStarted","Data":"a02d384c3bb2f0b294577263119369d5b275fc5a9f0f7f90bc60dec53935998a"}
Nov 02 01:32:26.872315 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:26.872213    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gws4h" event={"ID":"841f3428-10c6-4079-a256-cf6e5cd21806","Type":"ContainerStarted","Data":"a34c10891ac5eaa99d43919159eb3b66baf601fbd3d68dda19b525f82dffd5cd"}
Nov 02 01:32:26.991758 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:26.991254    2547 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gws4h" podStartSLOduration=67.228011938 podStartE2EDuration="1m8.991214731s" totalImagesPullingTime="1.763202793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:26.989677701 +0000 UTC m=+69.040993459" watchObservedRunningTime="2026-11-02 01:32:26.991214731 +0000 UTC m=+69.042530490"
Nov 02 01:32:27.882262 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:27.882051    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf" event={"ID":"c333281d-bd01-4c24-bb15-fa3ee2583fb9","Type":"ContainerStarted","Data":"1a2344340a58f718550f1fd797c6c14dd24832782377b5672ab708ff7c371cd4"}
Nov 02 01:32:27.882262 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:27.882123    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf" event={"ID":"c333281d-bd01-4c24-bb15-fa3ee2583fb9","Type":"ContainerStarted","Data":"2c5f3b22c5cc9fd472f5221dc28e995fe401a386ed45ec79976e3df03d7884d3"}
Nov 02 01:32:27.882262 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:27.882163    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf" event={"ID":"c333281d-bd01-4c24-bb15-fa3ee2583fb9","Type":"ContainerStarted","Data":"75bb3ca59f4bf75373fcea24fee69854f2c4f303c8ff3522c6fa640a87050b19"}
Nov 02 01:32:28.903453 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:28.903411    2547 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rc9qm"
Nov 02 01:32:28.908603 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:28.908578    2547 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rc9qm"
Nov 02 01:32:29.137972 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.137869    2547 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rc9qm" podStartSLOduration=2.140760185 podStartE2EDuration="5.137849361s" totalImagesPullingTime="2.997089176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:29.021193412 +0000 UTC m=+71.072509172" watchObservedRunningTime="2026-11-02 01:32:29.137849361 +0000 UTC m=+71.189165121"
Nov 02 01:32:29.564022 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.563988    2547 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 01:32:29.568021 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.567957    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.614594 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.614561    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.614722 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.614603    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.614722 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.614632    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.614722 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.614666    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.614722 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.614694    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31ccc613-71f9-40b5-a1f9-9254f9e7598b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.614904 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.614724    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.614904 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.614748    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc7c6\" (UniqueName: \"kubernetes.io/projected/31ccc613-71f9-40b5-a1f9-9254f9e7598b-kube-api-access-qc7c6\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.614904 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.614773    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-config\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.614904 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.614795    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31ccc613-71f9-40b5-a1f9-9254f9e7598b-config-out\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.614904 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.614815    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.614904 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.614847    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.614904 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.614883    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.614904 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.614904    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.615179 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.614930    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-web-config\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.615179 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.614963    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.615179 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.614984    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.615179 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.615021    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.615179 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.615060    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.615179 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.615100    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.615179 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.615135    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.622953 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.621443    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 02 01:32:29.622953 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.621739    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 02 01:32:29.622953 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.621786    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-22x2k\""
Nov 02 01:32:29.622953 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.621849    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 02 01:32:29.649271 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.648777    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 02 01:32:29.649271 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.648884    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 02 01:32:29.649271 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.649017    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 02 01:32:29.649271 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.649052    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 02 01:32:29.649740 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.649708    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 02 01:32:29.650283 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.650113    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 02 01:32:29.650283 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.650169    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 02 01:32:29.650283 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.650192    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 02 01:32:29.650283 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.650255    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-39jc004slgm3b\""
Nov 02 01:32:29.670741 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.670720    2547 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 02 01:32:29.715550 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.715470    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.715550 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.715505    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.715550 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.715524    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.715550 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.715547    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.715779 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.715570    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.715779 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.715590    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31ccc613-71f9-40b5-a1f9-9254f9e7598b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.715779 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.715611    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.715779 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.715634    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc7c6\" (UniqueName: \"kubernetes.io/projected/31ccc613-71f9-40b5-a1f9-9254f9e7598b-kube-api-access-qc7c6\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.715779 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.715651    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-config\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.715779 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.715673    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31ccc613-71f9-40b5-a1f9-9254f9e7598b-config-out\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.715779 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.715696    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.715779 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.715729    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.715779 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.715748    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.715779 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.715762    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.716083 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.715790    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-web-config\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.716083 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.715812    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.716083 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.715828    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.716083 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.715852    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.716083 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.715873    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.716083 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.715893    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.716476 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.716457    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.716984 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.716959    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.718887 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.718867    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.720442 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.719599    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.720442 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.720167    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.721211 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.720915    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-web-config\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.723974 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.723882    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.724079 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.724042    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.726249 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.725068    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.726249 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.725175    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.726249 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.725364    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31ccc613-71f9-40b5-a1f9-9254f9e7598b-config-out\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.726249 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.725509    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.726249 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.725962    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.727268 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.726810    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.730176 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.729769    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-config\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.731542 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.731489    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31ccc613-71f9-40b5-a1f9-9254f9e7598b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.733735 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.733717    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.749160 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.748844    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 02 01:32:29.749160 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.748882    2547 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 02 01:32:29.750684 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.750660    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.753400 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.753379    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.778117 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.778068    2547 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc7c6\" (UniqueName: \"kubernetes.io/projected/31ccc613-71f9-40b5-a1f9-9254f9e7598b-kube-api-access-qc7c6\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.801797 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.801770    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 01:32:29.904578 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.904495    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:29.916171 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.916142    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e0fdbea-232a-4a5e-abe4-3869da804c72","Type":"ContainerStarted","Data":"bb400c45770004d2dcf8ff380f2d6208c9ee8b829dd62fb3e53f63b4ea3d38c2"}
Nov 02 01:32:29.916302 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.916252    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e0fdbea-232a-4a5e-abe4-3869da804c72","Type":"ContainerStarted","Data":"5850bdfb4ead613094576a77f4f9ef7385687a0e13c29cf7693a6dc0ef2f8398"}
Nov 02 01:32:29.916302 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.916297    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e0fdbea-232a-4a5e-abe4-3869da804c72","Type":"ContainerStarted","Data":"ad46a2edabcddbd3e9f15ce4d9894d049a0d5aaf9c2fc2a0b08db8e3c062726d"}
Nov 02 01:32:29.916401 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.916337    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e0fdbea-232a-4a5e-abe4-3869da804c72","Type":"ContainerStarted","Data":"586659c0f70929d188e32418856330c66b4904ae6e1749d108262ec1268263ee"}
Nov 02 01:32:29.916401 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.916376    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e0fdbea-232a-4a5e-abe4-3869da804c72","Type":"ContainerStarted","Data":"857a6eb46b16acf31d19c387431a0d0aa1f82a690bd78fa7f93b6975f706bf4b"}
Nov 02 01:32:29.916502 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.916413    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e0fdbea-232a-4a5e-abe4-3869da804c72","Type":"ContainerStarted","Data":"e2e645da2c8ae758a8ea9547107b8f095f5c550c1e2f328b17054d0b976a5389"}
Nov 02 01:32:29.919602 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.919574    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rc9qm" event={"ID":"e9dd01bc-7209-4fe5-ac79-8efaa6c14232","Type":"ContainerStarted","Data":"725a212132086ba2d835f500b31bf712b8252ab1ba8ffaf5b63ca221c61444a0"}
Nov 02 01:32:29.923274 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.923239    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff" event={"ID":"544ee005-9680-4d16-ae14-a2633763bf10","Type":"ContainerStarted","Data":"70e7ddf194f10c1f27569dce6c8a0d771148b16d3c210b0a5ebb607669dbc4d5"}
Nov 02 01:32:29.927763 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.927741    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf" event={"ID":"c333281d-bd01-4c24-bb15-fa3ee2583fb9","Type":"ContainerStarted","Data":"e09c34b11c069c69de17266047e56ec8e0f4e8674602d1ba2eac0669ded1cfb6"}
Nov 02 01:32:29.927866 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.927798    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf" event={"ID":"c333281d-bd01-4c24-bb15-fa3ee2583fb9","Type":"ContainerStarted","Data":"c4cd7c9bd55a669149cb4fb65cfdfb3e60a5e01371de5f042123394dfbc4fbd1"}
Nov 02 01:32:29.927866 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.927825    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf" event={"ID":"c333281d-bd01-4c24-bb15-fa3ee2583fb9","Type":"ContainerStarted","Data":"ea635c173c7386de368e94be69ebc5383bcfc0ed89177bf9d9046a0e9943c8b0"}
Nov 02 01:32:29.928392 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:29.928374    2547 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:30.051613 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:30.050423    2547 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.967420723 podStartE2EDuration="8.050406782s" totalImagesPullingTime="5.066428479s" totalInitContainerRuntime="16.55758ms" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:30.048362721 +0000 UTC m=+72.099678501" watchObservedRunningTime="2026-11-02 01:32:30.050406782 +0000 UTC m=+72.101722542"
Nov 02 01:32:30.174092 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:30.173996    2547 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf" podStartSLOduration=2.9806383629999997 podStartE2EDuration="8.173980799s" totalImagesPullingTime="5.193342436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:30.172512497 +0000 UTC m=+72.223828282" watchObservedRunningTime="2026-11-02 01:32:30.173980799 +0000 UTC m=+72.225296557"
Nov 02 01:32:30.355570 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:30.355516    2547 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff" podStartSLOduration=2.05434245 podStartE2EDuration="5.355498502s" totalImagesPullingTime="3.301156052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:30.291611886 +0000 UTC m=+72.342927661" watchObservedRunningTime="2026-11-02 01:32:30.355498502 +0000 UTC m=+72.406814261"
Nov 02 01:32:30.356382 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:30.356282    2547 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 01:32:30.360434 ip-10-0-141-68 kubenswrapper[2547]: W1102 01:32:30.360401    2547 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ccc613_71f9_40b5_a1f9_9254f9e7598b.slice/crio-eebedc5242e32bfe4d2b763cd48f8cedf8ea1af380030a8cdd9f1cfc83993024 WatchSource:0}: Error finding container eebedc5242e32bfe4d2b763cd48f8cedf8ea1af380030a8cdd9f1cfc83993024: Status 404 returned error can't find the container with id eebedc5242e32bfe4d2b763cd48f8cedf8ea1af380030a8cdd9f1cfc83993024
Nov 02 01:32:30.934360 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:30.934326    2547 generic.go:356] "Generic (PLEG): container finished" podID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerID="21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262" exitCode=0
Nov 02 01:32:30.934770 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:30.934432    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31ccc613-71f9-40b5-a1f9-9254f9e7598b","Type":"ContainerDied","Data":"21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262"}
Nov 02 01:32:30.934770 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:30.934536    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31ccc613-71f9-40b5-a1f9-9254f9e7598b","Type":"ContainerStarted","Data":"eebedc5242e32bfe4d2b763cd48f8cedf8ea1af380030a8cdd9f1cfc83993024"}
Nov 02 01:32:30.939476 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:30.937901    2547 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gzq27" event={"ID":"6a2f0fa5-cbce-4695-aa38-e8bbeb720280","Type":"ContainerStarted","Data":"441e4b5d55e977e72190d478cddc10170d45f7832cd02888460a7c8eaccd151d"}
Nov 02 01:32:30.939476 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:30.937943    2547 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:32:31.178651 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:31.178543    2547 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gzq27" podStartSLOduration=66.731849466 podStartE2EDuration="1m13.178525524s" totalImagesPullingTime="6.446676058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:31.177947763 +0000 UTC m=+73.229263527" watchObservedRunningTime="2026-11-02 01:32:31.178525524 +0000 UTC m=+73.229841283"
Nov 02 01:32:31.950089 ip-10-0-141-68 kubenswrapper[2547]: I1102 01:32:31.950058    2547 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:32.496095 ip-10-0-141-68 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 01:32:32.508651 ip-10-0-141-68 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 01:32:32.508823 ip-10-0-141-68 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 01:32:32.508870 ip-10-0-141-68 systemd[1]: kubelet.service: Consumed 4.478s CPU time.
Nov 02 01:32:32.524773 ip-10-0-141-68 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 01:32:32.588349 ip-10-0-141-68 kubenswrapper[5875]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:32:32.588349 ip-10-0-141-68 kubenswrapper[5875]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 01:32:32.588349 ip-10-0-141-68 kubenswrapper[5875]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:32:32.588349 ip-10-0-141-68 kubenswrapper[5875]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:32:32.590420 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590338    5875 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:32:32.590420 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590419    5875 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:32:32.590420 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590422    5875 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:32:32.590517 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590426    5875 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:32:32.590517 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590430    5875 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:32:32.590517 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590433    5875 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:32:32.590517 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590436    5875 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:32:32.590517 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590438    5875 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:32:32.590517 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590441    5875 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:32:32.590517 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590444    5875 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:32:32.590517 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590446    5875 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:32:32.590517 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590449    5875 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:32:32.590517 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590451    5875 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:32:32.590517 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590454    5875 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:32:32.590517 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590457    5875 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:32:32.590517 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590459    5875 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:32:32.590517 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590462    5875 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:32:32.590517 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590465    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:32:32.590517 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590467    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:32:32.590517 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590470    5875 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:32:32.590517 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590472    5875 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:32:32.590517 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590475    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:32:32.590988 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590477    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:32:32.590988 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590480    5875 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:32:32.590988 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590484    5875 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:32:32.590988 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590488    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:32:32.590988 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590492    5875 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:32:32.590988 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590496    5875 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:32:32.590988 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590505    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:32:32.590988 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590508    5875 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:32:32.590988 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590511    5875 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:32:32.590988 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590514    5875 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:32:32.590988 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590517    5875 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:32:32.590988 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590519    5875 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:32:32.590988 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590522    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:32:32.590988 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590525    5875 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:32:32.590988 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590528    5875 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:32:32.590988 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590530    5875 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:32:32.590988 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590533    5875 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:32:32.590988 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590536    5875 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:32:32.590988 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590538    5875 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:32:32.591466 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590541    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:32:32.591466 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590543    5875 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:32:32.591466 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590546    5875 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:32:32.591466 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590548    5875 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:32:32.591466 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590550    5875 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:32:32.591466 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590553    5875 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:32:32.591466 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590555    5875 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:32:32.591466 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590558    5875 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:32:32.591466 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590560    5875 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:32:32.591466 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590563    5875 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:32:32.591466 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590565    5875 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:32:32.591466 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590567    5875 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:32:32.591466 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590570    5875 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:32:32.591466 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590573    5875 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:32:32.591466 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590576    5875 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:32:32.591466 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590579    5875 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:32:32.591466 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590581    5875 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:32:32.591466 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590585    5875 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:32:32.591466 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590587    5875 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:32:32.591909 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590590    5875 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:32:32.591909 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590593    5875 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:32:32.591909 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590596    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:32:32.591909 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590599    5875 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:32:32.591909 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590601    5875 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:32:32.591909 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590604    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:32:32.591909 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590606    5875 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:32:32.591909 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590608    5875 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:32:32.591909 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590611    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:32:32.591909 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590613    5875 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:32:32.591909 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590616    5875 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:32:32.591909 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590618    5875 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:32:32.591909 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590620    5875 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:32:32.591909 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590623    5875 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:32:32.591909 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590625    5875 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:32:32.591909 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590628    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:32:32.591909 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590630    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:32:32.591909 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590633    5875 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:32:32.591909 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590636    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:32:32.592372 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590638    5875 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:32:32.592372 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590640    5875 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:32:32.592372 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590643    5875 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:32:32.592372 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590646    5875 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:32:32.592372 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590649    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:32:32.592372 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590651    5875 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:32:32.592372 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590654    5875 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:32:32.592372 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590657    5875 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:32:32.592372 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590659    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:32:32.592372 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590662    5875 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:32:32.592372 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590665    5875 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:32:32.592372 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590667    5875 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:32:32.592372 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590670    5875 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:32:32.592372 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590673    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:32:32.592372 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590675    5875 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:32:32.592372 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590678    5875 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:32:32.592372 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590682    5875 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:32:32.592372 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590684    5875 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:32:32.592372 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590687    5875 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:32:32.592372 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590689    5875 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:32:32.592894 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590691    5875 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:32:32.592894 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590694    5875 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:32:32.592894 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590696    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:32:32.592894 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590698    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:32:32.592894 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590701    5875 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:32:32.592894 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590703    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:32:32.592894 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.590706    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:32:32.592894 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591491    5875 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:32:32.592894 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591501    5875 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:32:32.592894 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591505    5875 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:32:32.592894 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591508    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:32:32.592894 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591511    5875 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:32:32.592894 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591514    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:32:32.592894 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591516    5875 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:32:32.592894 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591520    5875 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:32:32.592894 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591523    5875 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:32:32.592894 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591526    5875 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:32:32.592894 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591529    5875 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:32:32.592894 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591532    5875 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:32:32.593381 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591535    5875 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:32:32.593381 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591537    5875 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:32:32.593381 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591540    5875 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:32:32.593381 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591542    5875 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:32:32.593381 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591545    5875 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:32:32.593381 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591548    5875 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:32:32.593381 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591551    5875 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:32:32.593381 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591553    5875 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:32:32.593381 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591556    5875 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:32:32.593381 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591559    5875 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:32:32.593381 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591562    5875 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:32:32.593381 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591564    5875 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:32:32.593381 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591567    5875 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:32:32.593381 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591569    5875 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:32:32.593381 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591571    5875 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:32:32.593381 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591574    5875 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:32:32.593381 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591577    5875 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:32:32.593381 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591581    5875 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:32:32.593381 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591584    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:32:32.593832 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591586    5875 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:32:32.593832 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591588    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:32:32.593832 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591591    5875 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:32:32.593832 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591594    5875 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:32:32.593832 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591597    5875 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:32:32.593832 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591600    5875 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:32:32.593832 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591602    5875 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:32:32.593832 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591606    5875 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:32:32.593832 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591609    5875 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:32:32.593832 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591611    5875 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:32:32.593832 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591614    5875 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:32:32.593832 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591616    5875 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:32:32.593832 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591619    5875 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:32:32.593832 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591621    5875 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:32:32.593832 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591624    5875 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:32:32.593832 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591626    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:32:32.593832 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591629    5875 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:32:32.593832 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591631    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:32:32.593832 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591633    5875 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:32:32.594320 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591636    5875 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:32:32.594320 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591638    5875 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:32:32.594320 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591641    5875 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:32:32.594320 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591643    5875 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:32:32.594320 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591646    5875 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:32:32.594320 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591649    5875 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:32:32.594320 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591652    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:32:32.594320 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591654    5875 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:32:32.594320 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591657    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:32:32.594320 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591659    5875 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:32:32.594320 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591662    5875 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:32:32.594320 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591665    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:32:32.594320 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591667    5875 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:32:32.594320 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591670    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:32:32.594320 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591672    5875 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:32:32.594320 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591675    5875 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:32:32.594320 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591677    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:32:32.594320 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591679    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:32:32.594320 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591682    5875 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:32:32.594811 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591684    5875 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:32:32.594811 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591687    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:32:32.594811 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591689    5875 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:32:32.594811 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591691    5875 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:32:32.594811 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591694    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:32:32.594811 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591696    5875 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:32:32.594811 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591699    5875 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:32:32.594811 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591701    5875 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:32:32.594811 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591704    5875 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:32:32.594811 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591706    5875 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:32:32.594811 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591708    5875 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:32:32.594811 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591711    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:32:32.594811 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591713    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:32:32.594811 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591715    5875 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:32:32.594811 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591718    5875 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:32:32.594811 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591720    5875 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:32:32.594811 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591722    5875 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:32:32.594811 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591725    5875 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:32:32.594811 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591728    5875 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:32:32.594811 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591730    5875 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:32:32.595310 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591733    5875 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:32:32.595310 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591735    5875 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:32:32.595310 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591737    5875 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:32:32.595310 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591740    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:32:32.595310 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591743    5875 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:32:32.595310 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591745    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:32:32.595310 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591748    5875 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:32:32.595310 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591750    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:32:32.595310 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591753    5875 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:32:32.595310 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591755    5875 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:32:32.595310 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591757    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:32:32.595310 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591760    5875 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:32:32.595310 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591762    5875 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:32:32.595310 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591764    5875 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:32:32.595310 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591767    5875 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:32:32.595310 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591769    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:32:32.595310 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.591772    5875 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:32:32.595310 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592292    5875 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:32:32.595310 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592298    5875 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:32:32.595762 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592301    5875 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:32:32.595762 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592304    5875 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:32:32.595762 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592307    5875 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:32:32.595762 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592309    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:32:32.595762 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592312    5875 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:32:32.595762 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592315    5875 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:32:32.595762 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592318    5875 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:32:32.595762 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592321    5875 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:32:32.595762 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592324    5875 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:32:32.595762 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592327    5875 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:32:32.595762 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592329    5875 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:32:32.595762 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592333    5875 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:32:32.595762 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592336    5875 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:32:32.595762 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592338    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:32:32.595762 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592341    5875 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:32:32.595762 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592343    5875 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:32:32.595762 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592346    5875 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:32:32.595762 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592350    5875 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:32:32.595762 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592354    5875 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:32:32.596285 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592357    5875 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:32:32.596285 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592360    5875 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:32:32.596285 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592363    5875 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:32:32.596285 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592366    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:32:32.596285 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592369    5875 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:32:32.596285 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592372    5875 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:32:32.596285 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592374    5875 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:32:32.596285 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592377    5875 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:32:32.596285 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592379    5875 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:32:32.596285 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592382    5875 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:32:32.596285 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592385    5875 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:32:32.596285 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592388    5875 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:32:32.596285 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592390    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:32:32.596285 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592392    5875 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:32:32.596285 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592395    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:32:32.596285 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592398    5875 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:32:32.596285 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592402    5875 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:32:32.596285 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592404    5875 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:32:32.596285 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592407    5875 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:32:32.597004 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592410    5875 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:32:32.597004 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592417    5875 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:32:32.597004 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592420    5875 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:32:32.597004 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592422    5875 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:32:32.597004 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592425    5875 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:32:32.597004 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592427    5875 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:32:32.597004 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592430    5875 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:32:32.597004 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592433    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:32:32.597004 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592436    5875 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:32:32.597004 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592439    5875 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:32:32.597004 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592442    5875 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:32:32.597004 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592444    5875 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:32:32.597004 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592447    5875 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:32:32.597004 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592449    5875 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:32:32.597004 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592451    5875 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:32:32.597004 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592454    5875 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:32:32.597004 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592456    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:32:32.597004 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592459    5875 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:32:32.597004 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592461    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:32:32.597637 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592463    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:32:32.597637 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592466    5875 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:32:32.597637 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592468    5875 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:32:32.597637 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592471    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:32:32.597637 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592473    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:32:32.597637 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592476    5875 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:32:32.597637 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592478    5875 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:32:32.597637 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592481    5875 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:32:32.597637 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592483    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:32:32.597637 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592485    5875 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:32:32.597637 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592488    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:32:32.597637 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592490    5875 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:32:32.597637 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592493    5875 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:32:32.597637 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592495    5875 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:32:32.597637 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592499    5875 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:32:32.597637 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592502    5875 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:32:32.597637 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592504    5875 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:32:32.597637 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592507    5875 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:32:32.597637 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592509    5875 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:32:32.597637 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592512    5875 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:32:32.598118 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592515    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:32:32.598118 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592517    5875 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:32:32.598118 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592520    5875 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:32:32.598118 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592522    5875 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:32:32.598118 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592525    5875 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:32:32.598118 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592527    5875 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:32:32.598118 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592530    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:32:32.598118 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592532    5875 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:32:32.598118 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592534    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:32:32.598118 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592537    5875 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:32:32.598118 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592540    5875 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:32:32.598118 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592542    5875 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:32:32.598118 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592545    5875 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:32:32.598118 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592547    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:32:32.598118 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592549    5875 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:32:32.598118 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592552    5875 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:32:32.598118 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592555    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:32:32.598118 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592557    5875 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:32:32.598118 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592559    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:32:32.598118 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592562    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:32:32.598650 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592564    5875 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:32:32.598650 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592567    5875 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:32:32.598650 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592569    5875 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:32:32.598650 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592572    5875 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:32:32.598650 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592574    5875 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:32:32.598650 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592576    5875 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:32:32.598650 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.592583    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:32:32.598650 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592673    5875 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 01:32:32.598650 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592685    5875 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 01:32:32.598650 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592691    5875 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 01:32:32.598650 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592696    5875 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 01:32:32.598650 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592700    5875 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 01:32:32.598650 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592704    5875 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 01:32:32.598650 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592708    5875 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 01:32:32.598650 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592713    5875 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 01:32:32.598650 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592716    5875 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 01:32:32.598650 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592719    5875 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 01:32:32.598650 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592722    5875 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 01:32:32.598650 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592725    5875 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 01:32:32.598650 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592728    5875 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 01:32:32.598650 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592731    5875 flags.go:64] FLAG: --cgroup-root=""
Nov 02 01:32:32.598650 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592734    5875 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 01:32:32.599182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592737    5875 flags.go:64] FLAG: --client-ca-file=""
Nov 02 01:32:32.599182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592740    5875 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 01:32:32.599182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592743    5875 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 01:32:32.599182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592747    5875 flags.go:64] FLAG: --cluster-domain=""
Nov 02 01:32:32.599182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592750    5875 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 01:32:32.599182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592753    5875 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 01:32:32.599182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592757    5875 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 01:32:32.599182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592760    5875 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 01:32:32.599182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592764    5875 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 01:32:32.599182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592766    5875 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 01:32:32.599182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592770    5875 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 01:32:32.599182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592773    5875 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 01:32:32.599182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592775    5875 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 01:32:32.599182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592778    5875 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 01:32:32.599182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592781    5875 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 01:32:32.599182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592784    5875 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 01:32:32.599182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592790    5875 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 01:32:32.599182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592794    5875 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 01:32:32.599182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592797    5875 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 01:32:32.599182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592800    5875 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 01:32:32.599182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592803    5875 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 01:32:32.599182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592806    5875 flags.go:64] FLAG: --enable-server="true"
Nov 02 01:32:32.599182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592809    5875 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 01:32:32.599182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592813    5875 flags.go:64] FLAG: --event-burst="100"
Nov 02 01:32:32.599182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592817    5875 flags.go:64] FLAG: --event-qps="50"
Nov 02 01:32:32.599791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592820    5875 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 01:32:32.599791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592823    5875 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 01:32:32.599791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592826    5875 flags.go:64] FLAG: --eviction-hard=""
Nov 02 01:32:32.599791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592830    5875 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 01:32:32.599791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592833    5875 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 01:32:32.599791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592835    5875 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 01:32:32.599791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592853    5875 flags.go:64] FLAG: --eviction-soft=""
Nov 02 01:32:32.599791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592858    5875 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 01:32:32.599791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592861    5875 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 01:32:32.599791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592864    5875 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 01:32:32.599791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592867    5875 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 01:32:32.599791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592870    5875 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 01:32:32.599791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592873    5875 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 01:32:32.599791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592877    5875 flags.go:64] FLAG: --feature-gates=""
Nov 02 01:32:32.599791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592881    5875 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 01:32:32.599791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592884    5875 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 01:32:32.599791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592887    5875 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 01:32:32.599791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592890    5875 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 01:32:32.599791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592893    5875 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 01:32:32.599791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592896    5875 flags.go:64] FLAG: --help="false"
Nov 02 01:32:32.599791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592899    5875 flags.go:64] FLAG: --hostname-override="ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:32:32.599791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592902    5875 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 01:32:32.599791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592905    5875 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 01:32:32.599791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592910    5875 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 01:32:32.600381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592913    5875 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 01:32:32.600381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592917    5875 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 01:32:32.600381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592920    5875 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 01:32:32.600381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592922    5875 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 01:32:32.600381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592925    5875 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 01:32:32.600381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592928    5875 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 01:32:32.600381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592931    5875 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 01:32:32.600381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592934    5875 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 01:32:32.600381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592937    5875 flags.go:64] FLAG: --kube-reserved=""
Nov 02 01:32:32.600381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592940    5875 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 01:32:32.600381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592943    5875 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 01:32:32.600381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592946    5875 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 01:32:32.600381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592949    5875 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 01:32:32.600381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592952    5875 flags.go:64] FLAG: --lock-file=""
Nov 02 01:32:32.600381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592954    5875 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 01:32:32.600381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592957    5875 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 01:32:32.600381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592960    5875 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 01:32:32.600381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592965    5875 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 01:32:32.600381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592968    5875 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 01:32:32.600381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592971    5875 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 01:32:32.600381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592974    5875 flags.go:64] FLAG: --logging-format="text"
Nov 02 01:32:32.600381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592977    5875 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 01:32:32.600381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592980    5875 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 01:32:32.600381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592983    5875 flags.go:64] FLAG: --manifest-url=""
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592986    5875 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592990    5875 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592993    5875 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592997    5875 flags.go:64] FLAG: --max-pods="110"
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.592999    5875 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593002    5875 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593005    5875 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593010    5875 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593013    5875 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593016    5875 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593018    5875 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593025    5875 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593029    5875 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593031    5875 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593034    5875 flags.go:64] FLAG: --pod-cidr=""
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593037    5875 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593040    5875 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593044    5875 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593047    5875 flags.go:64] FLAG: --port="10250"
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593050    5875 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593052    5875 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0dc07e14c333e45e9"
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593055    5875 flags.go:64] FLAG: --qos-reserved=""
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593058    5875 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593061    5875 flags.go:64] FLAG: --register-node="true"
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593064    5875 flags.go:64] FLAG: --register-with-taints=""
Nov 02 01:32:32.600946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593067    5875 flags.go:64] FLAG: --registry-burst="10"
Nov 02 01:32:32.601567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593071    5875 flags.go:64] FLAG: --registry-qps="5"
Nov 02 01:32:32.601567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593073    5875 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 01:32:32.601567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593076    5875 flags.go:64] FLAG: --reserved-memory=""
Nov 02 01:32:32.601567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593080    5875 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 01:32:32.601567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593083    5875 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 01:32:32.601567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593086    5875 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 01:32:32.601567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593089    5875 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 01:32:32.601567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593091    5875 flags.go:64] FLAG: --runonce="false"
Nov 02 01:32:32.601567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593094    5875 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 01:32:32.601567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593097    5875 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 01:32:32.601567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593101    5875 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 01:32:32.601567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593103    5875 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 01:32:32.601567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593106    5875 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 01:32:32.601567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593110    5875 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 01:32:32.601567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593113    5875 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 01:32:32.601567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593116    5875 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 01:32:32.601567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593119    5875 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 01:32:32.601567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593122    5875 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 01:32:32.601567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593124    5875 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 01:32:32.601567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593127    5875 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 01:32:32.601567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593130    5875 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 01:32:32.601567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593133    5875 flags.go:64] FLAG: --system-cgroups=""
Nov 02 01:32:32.601567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593136    5875 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 01:32:32.601567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593141    5875 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 01:32:32.601567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593144    5875 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 01:32:32.602158 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593147    5875 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 01:32:32.602158 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593151    5875 flags.go:64] FLAG: --tls-min-version=""
Nov 02 01:32:32.602158 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593154    5875 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 01:32:32.602158 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593157    5875 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 01:32:32.602158 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593160    5875 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 01:32:32.602158 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593162    5875 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 01:32:32.602158 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593165    5875 flags.go:64] FLAG: --v="2"
Nov 02 01:32:32.602158 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593169    5875 flags.go:64] FLAG: --version="false"
Nov 02 01:32:32.602158 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593173    5875 flags.go:64] FLAG: --vmodule=""
Nov 02 01:32:32.602158 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593177    5875 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 01:32:32.602158 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593180    5875 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 01:32:32.602158 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593334    5875 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:32:32.602158 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593338    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:32:32.602158 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593342    5875 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:32:32.602158 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593347    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:32:32.602158 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593350    5875 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:32:32.602158 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593353    5875 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:32:32.602158 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593356    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:32:32.602158 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593359    5875 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:32:32.602158 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593362    5875 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:32:32.602158 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593366    5875 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:32:32.602158 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593369    5875 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:32:32.602699 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593372    5875 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:32:32.602699 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593374    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:32:32.602699 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593377    5875 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:32:32.602699 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593379    5875 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:32:32.602699 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593382    5875 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:32:32.602699 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593384    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:32:32.602699 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593387    5875 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:32:32.602699 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593390    5875 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:32:32.602699 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593393    5875 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:32:32.602699 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593395    5875 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:32:32.602699 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593399    5875 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:32:32.602699 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593401    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:32:32.602699 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593404    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:32:32.602699 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593406    5875 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:32:32.602699 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593409    5875 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:32:32.602699 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593411    5875 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:32:32.602699 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593414    5875 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:32:32.602699 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593416    5875 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:32:32.602699 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593418    5875 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:32:32.603216 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593421    5875 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:32:32.603216 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593424    5875 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:32:32.603216 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593428    5875 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:32:32.603216 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593430    5875 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:32:32.603216 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593433    5875 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:32:32.603216 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593435    5875 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:32:32.603216 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593437    5875 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:32:32.603216 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593440    5875 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:32:32.603216 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593442    5875 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:32:32.603216 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593445    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:32:32.603216 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593447    5875 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:32:32.603216 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593451    5875 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:32:32.603216 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593453    5875 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:32:32.603216 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593456    5875 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:32:32.603216 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593459    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:32:32.603216 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593461    5875 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:32:32.603216 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593464    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:32:32.603216 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593466    5875 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:32:32.603216 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593469    5875 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:32:32.603216 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593471    5875 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:32:32.603804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593475    5875 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:32:32.603804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593479    5875 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:32:32.603804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593482    5875 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:32:32.603804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593485    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:32:32.603804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593488    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:32:32.603804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593491    5875 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:32:32.603804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593493    5875 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:32:32.603804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593495    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:32:32.603804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593498    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:32:32.603804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593500    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:32:32.603804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593503    5875 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:32:32.603804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593505    5875 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:32:32.603804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593508    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:32:32.603804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593510    5875 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:32:32.603804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593514    5875 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:32:32.603804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593516    5875 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:32:32.603804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593519    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:32:32.603804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593521    5875 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:32:32.603804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593523    5875 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:32:32.604312 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593526    5875 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:32:32.604312 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593528    5875 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:32:32.604312 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593531    5875 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:32:32.604312 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593533    5875 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:32:32.604312 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593537    5875 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:32:32.604312 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593539    5875 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:32:32.604312 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593542    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:32:32.604312 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593544    5875 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:32:32.604312 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593546    5875 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:32:32.604312 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593549    5875 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:32:32.604312 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593551    5875 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:32:32.604312 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593554    5875 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:32:32.604312 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593556    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:32:32.604312 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593558    5875 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:32:32.604312 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593561    5875 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:32:32.604312 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593563    5875 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:32:32.604312 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593566    5875 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:32:32.604312 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593568    5875 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:32:32.604312 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593571    5875 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:32:32.604312 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593574    5875 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:32:32.605034 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593577    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:32:32.605034 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593579    5875 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:32:32.605034 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593582    5875 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:32:32.605034 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593584    5875 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:32:32.605034 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593586    5875 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:32:32.605034 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593589    5875 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:32:32.605034 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593591    5875 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:32:32.605034 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593595    5875 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:32:32.605034 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593597    5875 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:32:32.605034 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593600    5875 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:32:32.605034 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593602    5875 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:32:32.605034 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593605    5875 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:32:32.605034 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593607    5875 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:32:32.605034 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593610    5875 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:32:32.605034 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593612    5875 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:32:32.605034 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593615    5875 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:32:32.605034 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593619    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:32:32.605804 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.593646    5875 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:32:32.605804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593684    5875 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:32:32.605804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593689    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:32:32.605804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593691    5875 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:32:32.605804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593694    5875 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:32:32.605804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593697    5875 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:32:32.605804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593699    5875 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:32:32.605804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593702    5875 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:32:32.605804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593705    5875 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:32:32.605804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593707    5875 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:32:32.605804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593710    5875 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:32:32.605804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593712    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:32:32.605804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593715    5875 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:32:32.605804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593717    5875 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:32:32.605804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593720    5875 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:32:32.605804 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593723    5875 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:32:32.606521 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593725    5875 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:32:32.606521 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593728    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:32:32.606521 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593730    5875 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:32:32.606521 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593733    5875 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:32:32.606521 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593736    5875 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:32:32.606521 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593738    5875 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:32:32.606521 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593742    5875 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:32:32.606521 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593744    5875 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:32:32.606521 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593747    5875 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:32:32.606521 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593749    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:32:32.606521 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593751    5875 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:32:32.606521 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593754    5875 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:32:32.606521 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593756    5875 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:32:32.606521 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593759    5875 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:32:32.606521 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593761    5875 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:32:32.606521 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593764    5875 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:32:32.606521 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593767    5875 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:32:32.606521 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593770    5875 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:32:32.606521 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593773    5875 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:32:32.606521 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593775    5875 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:32:32.607177 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593778    5875 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:32:32.607177 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593780    5875 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:32:32.607177 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593783    5875 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:32:32.607177 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593785    5875 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:32:32.607177 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593787    5875 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:32:32.607177 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593790    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:32:32.607177 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593792    5875 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:32:32.607177 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593795    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:32:32.607177 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593798    5875 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:32:32.607177 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593802    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:32:32.607177 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593805    5875 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:32:32.607177 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593808    5875 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:32:32.607177 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593811    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:32:32.607177 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593813    5875 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:32:32.607177 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593815    5875 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:32:32.607177 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593818    5875 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:32:32.607177 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593821    5875 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:32:32.607177 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593823    5875 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:32:32.607177 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593827    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:32:32.607913 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593829    5875 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:32:32.607913 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593832    5875 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:32:32.607913 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593834    5875 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:32:32.607913 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593837    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:32:32.607913 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593839    5875 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:32:32.607913 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593842    5875 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:32:32.607913 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593844    5875 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:32:32.607913 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593847    5875 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:32:32.607913 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593849    5875 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:32:32.607913 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593853    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:32:32.607913 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593856    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:32:32.607913 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593858    5875 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:32:32.607913 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593861    5875 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:32:32.607913 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593863    5875 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:32:32.607913 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593865    5875 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:32:32.607913 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593868    5875 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:32:32.607913 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593870    5875 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:32:32.607913 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593873    5875 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:32:32.607913 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593875    5875 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:32:32.608426 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593877    5875 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:32:32.608426 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593880    5875 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:32:32.608426 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593883    5875 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:32:32.608426 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593885    5875 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:32:32.608426 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593887    5875 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:32:32.608426 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593890    5875 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:32:32.608426 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593893    5875 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:32:32.608426 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593896    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:32:32.608426 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593898    5875 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:32:32.608426 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593901    5875 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:32:32.608426 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593903    5875 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:32:32.608426 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593906    5875 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:32:32.608426 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593909    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:32:32.608426 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593912    5875 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:32:32.608426 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593914    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:32:32.608426 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593916    5875 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:32:32.608426 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593919    5875 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:32:32.608426 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593921    5875 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:32:32.608426 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593926    5875 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:32:32.608426 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593930    5875 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:32:32.608906 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593933    5875 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:32:32.608906 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593936    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:32:32.608906 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593940    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:32:32.608906 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593943    5875 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:32:32.608906 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593945    5875 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:32:32.608906 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593947    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:32:32.608906 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593950    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:32:32.608906 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593952    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:32:32.608906 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593955    5875 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:32:32.608906 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593957    5875 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:32:32.608906 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593960    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:32:32.608906 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593962    5875 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:32:32.608906 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.593964    5875 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:32:32.608906 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.598832    5875 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 01:32:32.608906 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.598849    5875 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 01:32:32.608906 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.598868    5875 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 01:32:32.608906 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.598873    5875 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 01:32:32.608906 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.598949    5875 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:32:32.608906 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.598955    5875 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:32:32.609374 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.598959    5875 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:32:32.609374 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.598962    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:32:32.609374 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.598964    5875 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:32:32.609374 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.598967    5875 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:32:32.609374 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.598970    5875 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:32:32.609374 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.598973    5875 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:32:32.609374 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.598975    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:32:32.609374 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.598978    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:32:32.609374 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.598980    5875 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:32:32.609374 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.598983    5875 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:32:32.609374 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.598985    5875 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:32:32.609374 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.598988    5875 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:32:32.609374 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.598991    5875 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:32:32.609374 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.598994    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:32:32.609374 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.598996    5875 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:32:32.609374 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.598999    5875 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:32:32.609374 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599001    5875 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:32:32.609374 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599004    5875 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:32:32.609374 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599008    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:32:32.609374 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599010    5875 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:32:32.609837 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599013    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:32:32.609837 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599016    5875 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:32:32.609837 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599019    5875 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:32:32.609837 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599021    5875 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:32:32.609837 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599028    5875 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:32:32.609837 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599031    5875 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:32:32.609837 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599034    5875 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:32:32.609837 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599036    5875 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:32:32.609837 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599039    5875 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:32:32.609837 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599041    5875 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:32:32.609837 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599044    5875 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:32:32.609837 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599048    5875 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:32:32.609837 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599052    5875 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:32:32.609837 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599054    5875 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:32:32.609837 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599057    5875 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:32:32.609837 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599059    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:32:32.609837 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599062    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:32:32.609837 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599064    5875 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:32:32.609837 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599067    5875 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:32:32.610340 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599070    5875 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:32:32.610340 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599072    5875 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:32:32.610340 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599075    5875 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:32:32.610340 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599077    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:32:32.610340 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599080    5875 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:32:32.610340 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599083    5875 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:32:32.610340 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599086    5875 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:32:32.610340 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599089    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:32:32.610340 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599092    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:32:32.610340 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599094    5875 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:32:32.610340 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599097    5875 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:32:32.610340 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599099    5875 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:32:32.610340 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599102    5875 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:32:32.610340 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599105    5875 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:32:32.610340 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599107    5875 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:32:32.610340 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599110    5875 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:32:32.610340 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599112    5875 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:32:32.610340 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599115    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:32:32.610340 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599117    5875 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:32:32.610340 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599121    5875 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:32:32.610893 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599125    5875 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:32:32.610893 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599128    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:32:32.610893 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599131    5875 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:32:32.610893 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599134    5875 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:32:32.610893 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599136    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:32:32.610893 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599139    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:32:32.610893 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599141    5875 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:32:32.610893 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599144    5875 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:32:32.610893 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599147    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:32:32.610893 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599149    5875 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:32:32.610893 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599152    5875 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:32:32.610893 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599155    5875 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:32:32.610893 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599157    5875 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:32:32.610893 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599160    5875 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:32:32.610893 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599162    5875 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:32:32.610893 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599165    5875 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:32:32.610893 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599167    5875 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:32:32.610893 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599170    5875 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:32:32.610893 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599173    5875 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:32:32.610893 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599176    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:32:32.611562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599178    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:32:32.611562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599181    5875 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:32:32.611562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599183    5875 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:32:32.611562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599186    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:32:32.611562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599189    5875 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:32:32.611562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599191    5875 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:32:32.611562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599194    5875 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:32:32.611562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599197    5875 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:32:32.611562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599200    5875 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:32:32.611562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599202    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:32:32.611562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599205    5875 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:32:32.611562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599207    5875 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:32:32.611562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599209    5875 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:32:32.611562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599212    5875 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:32:32.611562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599214    5875 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:32:32.611562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599230    5875 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:32:32.611562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599233    5875 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:32:32.611562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599236    5875 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:32:32.611562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599239    5875 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:32:32.612156 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599241    5875 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:32:32.612156 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599243    5875 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:32:32.612156 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599246    5875 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:32:32.612156 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599248    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:32:32.612156 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599251    5875 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:32:32.612156 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599253    5875 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:32:32.612156 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.599280    5875 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:32:32.612156 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599427    5875 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:32:32.612156 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599432    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:32:32.612156 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599436    5875 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:32:32.612156 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599439    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:32:32.612156 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599443    5875 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:32:32.612156 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599447    5875 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:32:32.612156 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599450    5875 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:32:32.612156 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599453    5875 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:32:32.612156 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599456    5875 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:32:32.612889 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599458    5875 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:32:32.612889 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599461    5875 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:32:32.612889 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599464    5875 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:32:32.612889 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599466    5875 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:32:32.612889 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599469    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:32:32.612889 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599471    5875 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:32:32.612889 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599473    5875 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:32:32.612889 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599476    5875 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:32:32.612889 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599478    5875 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:32:32.612889 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599481    5875 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:32:32.612889 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599483    5875 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:32:32.612889 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599486    5875 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:32:32.612889 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599489    5875 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:32:32.612889 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599491    5875 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:32:32.612889 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599494    5875 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:32:32.612889 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599496    5875 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:32:32.612889 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599499    5875 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:32:32.612889 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599501    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:32:32.612889 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599503    5875 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:32:32.612889 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599506    5875 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:32:32.613546 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599509    5875 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:32:32.613546 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599512    5875 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:32:32.613546 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599514    5875 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:32:32.613546 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599516    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:32:32.613546 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599519    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:32:32.613546 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599521    5875 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:32:32.613546 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599524    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:32:32.613546 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599526    5875 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:32:32.613546 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599528    5875 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:32:32.613546 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599531    5875 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:32:32.613546 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599534    5875 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:32:32.613546 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599536    5875 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:32:32.613546 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599539    5875 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:32:32.613546 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599541    5875 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:32:32.613546 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599544    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:32:32.613546 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599546    5875 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:32:32.613546 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599549    5875 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:32:32.613546 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599551    5875 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:32:32.613546 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599554    5875 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:32:32.613546 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599556    5875 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:32:32.613546 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599559    5875 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:32:32.614119 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599562    5875 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:32:32.614119 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599564    5875 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:32:32.614119 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599567    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:32:32.614119 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599570    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:32:32.614119 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599572    5875 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:32:32.614119 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599574    5875 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:32:32.614119 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599577    5875 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:32:32.614119 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599579    5875 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:32:32.614119 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599582    5875 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:32:32.614119 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599584    5875 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:32:32.614119 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599586    5875 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:32:32.614119 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599588    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:32:32.614119 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599591    5875 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:32:32.614119 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599593    5875 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:32:32.614119 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599596    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:32:32.614119 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599598    5875 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:32:32.614119 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599601    5875 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:32:32.614119 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599603    5875 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:32:32.614119 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599606    5875 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:32:32.614119 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599608    5875 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:32:32.614624 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599610    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:32:32.614624 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599614    5875 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:32:32.614624 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599616    5875 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:32:32.614624 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599619    5875 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:32:32.614624 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599621    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:32:32.614624 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599623    5875 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:32:32.614624 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599626    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:32:32.614624 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599629    5875 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:32:32.614624 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599631    5875 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:32:32.614624 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599634    5875 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:32:32.614624 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599636    5875 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:32:32.614624 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599638    5875 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:32:32.614624 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599642    5875 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:32:32.614624 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599645    5875 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:32:32.614624 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599649    5875 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:32:32.614624 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599652    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:32:32.614624 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599654    5875 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:32:32.614624 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599656    5875 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:32:32.614624 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599659    5875 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:32:32.615408 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599661    5875 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:32:32.615408 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599664    5875 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:32:32.615408 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599666    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:32:32.615408 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599669    5875 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:32:32.615408 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599671    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:32:32.615408 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599674    5875 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:32:32.615408 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599676    5875 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:32:32.615408 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599679    5875 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:32:32.615408 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599682    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:32:32.615408 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599684    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:32:32.615408 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599686    5875 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:32:32.615408 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599689    5875 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:32:32.615408 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599692    5875 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:32:32.615408 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599694    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:32:32.615408 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599698    5875 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:32:32.615408 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599702    5875 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:32:32.615408 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599705    5875 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:32:32.616117 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.599727    5875 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:32:32.616117 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599757    5875 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:32:32.616117 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599761    5875 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:32:32.616117 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599764    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:32:32.616117 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599767    5875 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:32:32.616117 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599769    5875 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:32:32.616117 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599773    5875 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:32:32.616117 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599775    5875 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:32:32.616117 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599778    5875 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:32:32.616117 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599780    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:32:32.616117 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599783    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:32:32.616117 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599785    5875 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:32:32.616117 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599788    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:32:32.616117 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599790    5875 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:32:32.616117 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599793    5875 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:32:32.616117 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599796    5875 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:32:32.616562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599799    5875 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:32:32.616562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599802    5875 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:32:32.616562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599804    5875 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:32:32.616562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599807    5875 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:32:32.616562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599809    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:32:32.616562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599812    5875 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:32:32.616562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599815    5875 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:32:32.616562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599817    5875 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:32:32.616562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599819    5875 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:32:32.616562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599822    5875 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:32:32.616562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599825    5875 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:32:32.616562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599828    5875 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:32:32.616562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599830    5875 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:32:32.616562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599833    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:32:32.616562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599835    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:32:32.616562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599838    5875 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:32:32.616562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599841    5875 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:32:32.616562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599843    5875 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:32:32.616562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599845    5875 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:32:32.616562 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599848    5875 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:32:32.617047 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599850    5875 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:32:32.617047 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599852    5875 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:32:32.617047 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599855    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:32:32.617047 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599857    5875 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:32:32.617047 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599859    5875 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:32:32.617047 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599863    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:32:32.617047 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599865    5875 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:32:32.617047 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599867    5875 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:32:32.617047 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599870    5875 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:32:32.617047 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599872    5875 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:32:32.617047 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599874    5875 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:32:32.617047 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599877    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:32:32.617047 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599879    5875 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:32:32.617047 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599882    5875 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:32:32.617047 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599884    5875 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:32:32.617047 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599886    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:32:32.617047 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599889    5875 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:32:32.617047 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599891    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:32:32.617047 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599893    5875 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:32:32.617047 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599896    5875 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:32:32.617591 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599898    5875 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:32:32.617591 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599900    5875 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:32:32.617591 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599903    5875 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:32:32.617591 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599907    5875 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:32:32.617591 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599910    5875 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:32:32.617591 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599912    5875 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:32:32.617591 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599915    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:32:32.617591 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599917    5875 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:32:32.617591 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599920    5875 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:32:32.617591 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599923    5875 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:32:32.617591 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599925    5875 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:32:32.617591 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599928    5875 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:32:32.617591 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599930    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:32:32.617591 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599933    5875 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:32:32.617591 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599935    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:32:32.617591 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599937    5875 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:32:32.617591 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599940    5875 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:32:32.617591 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599942    5875 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:32:32.617591 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599945    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:32:32.618041 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599947    5875 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:32:32.618041 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599950    5875 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:32:32.618041 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599952    5875 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:32:32.618041 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599955    5875 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:32:32.618041 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599957    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:32:32.618041 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599961    5875 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:32:32.618041 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599964    5875 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:32:32.618041 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599967    5875 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:32:32.618041 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599970    5875 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:32:32.618041 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599972    5875 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:32:32.618041 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599974    5875 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:32:32.618041 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599977    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:32:32.618041 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599979    5875 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:32:32.618041 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599981    5875 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:32:32.618041 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599984    5875 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:32:32.618041 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599986    5875 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:32:32.618041 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599989    5875 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:32:32.618041 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599991    5875 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:32:32.618041 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599993    5875 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:32:32.618643 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599996    5875 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:32:32.618643 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.599998    5875 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:32:32.618643 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.600001    5875 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:32:32.618643 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.600004    5875 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:32:32.618643 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.600006    5875 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:32:32.618643 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.600009    5875 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:32:32.618643 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.600011    5875 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:32:32.618643 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.600013    5875 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:32:32.618643 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.600018    5875 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:32:32.618643 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.600020    5875 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:32:32.618643 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.600023    5875 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:32:32.618643 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.600025    5875 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:32:32.618643 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:32:32.600027    5875 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:32:32.618643 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.600075    5875 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 01:32:32.618643 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.601036    5875 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 01:32:32.618643 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.601100    5875 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 01:32:32.618643 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.601310    5875 server.go:1014] "Starting client certificate rotation"
Nov 02 01:32:32.618643 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.601449    5875 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 01:32:32.619083 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.601503    5875 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 01:26:18 +0000 UTC" deadline="2028-07-20 04:24:47.101960058 +0000 UTC"
Nov 02 01:32:32.619083 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.601530    5875 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15026h52m14.500431885s"
Nov 02 01:32:32.619083 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.601802    5875 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 01:32:32.619083 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.602877    5875 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 01:32:32.619083 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.604105    5875 log.go:25] "Validated CRI v1 runtime API"
Nov 02 01:32:32.619083 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.606703    5875 log.go:25] "Validated CRI v1 image API"
Nov 02 01:32:32.619083 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.607531    5875 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 01:32:32.619083 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.609831    5875 fs.go:135] Filesystem UUIDs: map[046c2f35-84f7-4bb3-8e7e-7c290fbde1de:/dev/nvme0n1p3 522aea20-40da-41ac-bac0-7a7a80fd7396:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 02 01:32:32.619334 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.609850    5875 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/069fc7fc54846606f1108d55c7f501f0612a442463133eb380c50e044f2b5eef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/069fc7fc54846606f1108d55c7f501f0612a442463133eb380c50e044f2b5eef/userdata/shm major:0 minor:370 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0db0a4440d26849b6911b476f8ddda7e31933f0ea3721836115dae07380692b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0db0a4440d26849b6911b476f8ddda7e31933f0ea3721836115dae07380692b7/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/173c5a4c499cae3df6ed358eb93376aa21a1fa387ba8ee04b6aad1f8dab9f9cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/173c5a4c499cae3df6ed358eb93376aa21a1fa387ba8ee04b6aad1f8dab9f9cb/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a39fcc2e1764800b057926da3ca002baee535c03f38487c841ddb9afed6adeb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a39fcc2e1764800b057926da3ca002baee535c03f38487c841ddb9afed6adeb/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/212349606df6c854d29a533d0079cf270f94912312bbddc935e656a210c5b596/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/212349606df6c854d29a533d0079cf270f94912312bbddc935e656a210c5b596/userdata/shm major:0 minor:376 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31a3b3106c6b4575f445dafbec4c249d38b1223bef155785bb0ea2faf6ec9f5c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31a3b3106c6b4575f445dafbec4c249d38b1223bef155785bb0ea2faf6ec9f5c/userdata/shm major:0 minor:264 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/374a8710e7ed9580be3a4b357b8d10b27f6d9121cb57ac4c27d1e0737cdcf3fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/374a8710e7ed9580be3a4b357b8d10b27f6d9121cb57ac4c27d1e0737cdcf3fc/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3a1f4347a710d8d6bf041568b564311aa0529b25267be62fb4e42109c6a0afae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3a1f4347a710d8d6bf041568b564311aa0529b25267be62fb4e42109c6a0afae/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/56b9dbaa2578f01ef84a655f37e6ee1c3a1c3602959ecb6b552615b7300c9990/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/56b9dbaa2578f01ef84a655f37e6ee1c3a1c3602959ecb6b552615b7300c9990/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/69ae9b34a115e17992488ab6bb6a978d66e0bdaf4fdb424715e2eee95957142d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/69ae9b34a115e17992488ab6bb6a978d66e0bdaf4fdb424715e2eee95957142d/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/746e119d52d73097b35d0921c93fddceeafd18cdc55e96cb368f9b9a5c54d5be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/746e119d52d73097b35d0921c93fddceeafd18cdc55e96cb368f9b9a5c54d5be/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74a59b76abefddb3b42ae880ccfb8df7538d03ad03fee714c400b67133f82159/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74a59b76abefddb3b42ae880ccfb8df7538d03ad03fee714c400b67133f82159/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7dba8084753fc3d3c7fb38ca22890cf10dd689e50a76867c80541fd8ee48baf8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7dba8084753fc3d3c7fb38ca22890cf10dd689e50a76867c80541fd8ee48baf8/userdata/shm major:0 minor:375 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e005dea061ad10babcd2a270d689a9cb0c2b5c78f224d8c1328fd08fda7c2e0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e005dea061ad10babcd2a270d689a9cb0c2b5c78f224d8c1328fd08fda7c2e0/userdata/shm major:0 minor:395 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8181c311e1c1935dc496d928f2500bd28e19de558f63de49bf74791c75f5bb03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8181c311e1c1935dc496d928f2500bd28e19de558f63de49bf74791c75f5bb03/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f9a4968a094313736ea605c9a4f7a4883f0786b353a37a3e88d8c9fa38ad501/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f9a4968a094313736ea605c9a4f7a4883f0786b353a37a3e88d8c9fa38ad501/userdata/shm major:0 minor:392 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94511ea34e79b2676f7c1c9811db65435d12c7d56f939ce4661f88d3f3ff18ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94511ea34e79b2676f7c1c9811db65435d12c7d56f939ce4661f88d3f3ff18ef/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9662347313a235d1d0e72eb393df4285943e8fbfe4cee5d50247970837213a04/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9662347313a235d1d0e72eb393df4285943e8fbfe4cee5d50247970837213a04/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ea0457b904929e89ca2f6b5468317750d056f07850e062ed00d655a5c8ad218/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ea0457b904929e89ca2f6b5468317750d056f07850e062ed00d655a5c8ad218/userdata/shm major:0 minor:285 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b178ceb687ed740a6de9b4f920ff574b54f851f0634ea61f239e6902bce2af03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b178ceb687ed740a6de9b4f920ff574b54f851f0634ea61f239e6902bce2af03/userdata/shm major:0 minor:321 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6d333f27e662dda79ef0693c4220289ca475279c9948cae04af48d4ce3e68ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6d333f27e662dda79ef0693c4220289ca475279c9948cae04af48d4ce3e68ac/userdata/shm major:0 minor:364 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b70d62c7a50f16a630ac3255032d18f9af3d701854c30ba24fddf6128ca24204/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b70d62c7a50f16a630ac3255032d18f9af3d701854c30ba24fddf6128ca24204/userdata/shm major:0 minor:371 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b93ed80f94e779453d490ac6563b3bc4fe54645d777edb0179458eb585a87bb1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b93ed80f94e779453d490ac6563b3bc4fe54645d777edb0179458eb585a87bb1/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e98988375935e3f025b17c237709d84c2173ddc0314bbecd7df3a7c758006e8d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e98988375935e3f025b17c237709d84c2173ddc0314bbecd7df3a7c758006e8d/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eebedc5242e32bfe4d2b763cd48f8cedf8ea1af380030a8cdd9f1cfc83993024/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eebedc5242e32bfe4d2b763cd48f8cedf8ea1af380030a8cdd9f1cfc83993024/userdata/shm major:0 minor:495 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f65a11b3eb88e9616d6d3b20d732fe322d46788e9eb4fea4e0de2f2c967838c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f65a11b3eb88e9616d6d3b20d732fe322d46788e9eb4fea4e0de2f2c967838c5/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9ea61f3a326dc5e719210e4d5309b24a1bbcad0df54b4d76cb0258695ad9641/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9ea61f3a326dc5e719210e4d5309b24a1bbcad0df54b4d76cb0258695ad9641/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/046dd898-c779-4ce2-84ec-dc91afb8323f/volumes/kubernetes.io~projected/kube-api-access-nnbp7:{mountpoint:/var/lib/kubelet/pods/046dd898-c779-4ce2-84ec-dc91afb8323f/volumes/kubernetes.io~projected/kube-api-access-nnbp7 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volume-subpaths/web-config/alertmanager/9 major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volume-subpaths/web-config/config-reloader/6 major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~empty-dir/config-out major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~projected/kube-api-access-gpswh:{mountpoint:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~projected/kube-api-access-gpswh major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~projected/tls-assets major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~secret/config-volume major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~secret/web-config major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59/volumes/kubernetes.io~projected/kube-api-access-jt4vp:{mountpoint:/var/lib/kubelet/pods/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59/volumes/kubernetes.io~projected/kube-api-access-jt4vp major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:483 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~empty-dir/config-out major:0 minor:487 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~projected/kube-api-access-qc7c6:{mountpoint:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~projected/kube-api-access-qc7c6 major:0 minor:478 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~projected/tls-assets major:0 minor:491 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/config major:0 minor:484 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:488 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:490 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:492 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:485 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:489 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:486 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:482 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/web-config major:0 minor:483 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20/volumes/kubernetes.io~projected/kube-api-access-zb7sn:{mountpoint:/var/lib/kubelet/pods/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20/volumes/kubernetes.io~projected/kube-api-access-zb7sn major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/544ee005-9680-4d16-ae14-a2633763bf10/volumes/kubernetes.io~projected/kube-api-access-rdrcw:{mountpoint:/var/lib/kubelet/pods/544ee005-9680-4d16-ae14-a2633763bf10/volumes/kubernetes.io~projected/kube-api-access-rdrcw major:0 minor:394 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/544ee005-9680-4d16-ae14-a2633763bf10/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/544ee005-9680-4d16-ae14-a2633763bf10/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:389 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/544ee005-9680-4d16-ae14-a2633763bf10/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/544ee005-9680-4d16-ae14-a2633763bf10/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:390 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/544ee005-9680-4d16-ae14-a2633763bf10/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/544ee005-9680-4d16-ae14-a2633763bf10/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:391 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5dbcf879-26f8-4f7b-a071-eafa688b1c05/volumes/kubernetes.io~projected/kube-api-access-7kjnt:{mountpoint:/var/lib/kubelet/pods/5dbcf879-26f8-4f7b-a071-eafa688b1c05/volumes/kubernetes.io~projected/kube-api-access-7kjnt major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f70370d-3793-4e25-90e8-fc252f5a7596/volumes/kubernetes.io~projected/kube-api-access-8hl42:{mountpoint:/var/lib/kubelet/pods/5f70370d-3793-4e25-90e8-fc252f5a7596/volumes/kubernetes.io~projected/kube-api-access-8hl42 major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f70370d-3793-4e25-90e8-fc252f5a7596/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/5f70370d-3793-4e25-90e8-fc252f5a7596/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f70370d-3793-4e25-90e8-fc252f5a7596/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/5f70370d-3793-4e25-90e8-fc252f5a7596/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65f63e43-859a-4d3a-8393-99724782596b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/65f63e43-859a-4d3a-8393-99724782596b/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a2f0fa5-cbce-4695-aa38-e8bbeb720280/volumes/kubernetes.io~projected/kube-api-access-c9sj2:{mountpoint:/var/lib/kubelet/pods/6a2f0fa5-cbce-4695-aa38-e8bbeb720280/volumes/kubernetes.io~projected/kube-api-access-c9sj2 major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e02ef8d-7b7b-4de9-a2db-ae32d200a497/volumes/kubernetes.io~projected/kube-api-access-4f4gw:{mountpoint:/var/lib/kubelet/pods/6e02ef8d-7b7b-4de9-a2db-ae32d200a497/volumes/kubernetes.io~projected/kube-api-access-4f4gw major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/792b1392-5f44-47a8-9b86-05ee811046ea/volumes/kubernetes.io~projected/kube-api-access-pkc2j:{mountpoint:/var/lib/kubelet/pods/792b1392-5f44-47a8-9b86-05ee811046ea/volumes/kubernetes.io~projected/kube-api-access-pkc2j major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/792b1392-5f44-47a8-9b86-05ee811046ea/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/792b1392-5f44-47a8-9b86-05ee811046ea/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/792b1392-5f44-47a8-9b86-05ee811046ea/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/792b1392-5f44-47a8-9b86-05ee811046ea/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/841f3428-10c6-4079-a256-cf6e5cd21806/volumes/kubernetes.io~projected/kube-api-access-2nd6c:{mountpoint:/var/lib/kubelet/pods/841f3428-10c6-4079-a256-cf6e5cd21806/volumes/kubernetes.io~projected/kube-api-access-2nd6c major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/841f3428-10c6-4079-a256-cf6e5cd21806/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/841f3428-10c6-4079-a256-cf6e5cd21806/volumes/kubernetes.io~secret/metrics-certs major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85999952-cf96-48d1-a608-91c84a7bc17c/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/85999952-cf96-48d1-a608-91c84a7bc17c/volumes/kubernetes.io~secret/tls-certificates major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28/volumes/kubernetes.io~projected/kube-api-access-5vmxb:{mountpoint:/var/lib/kubelet/pods/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28/volumes/kubernetes.io~projected/kube-api-access-5vmxb major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/902d51a8-2407-4f4b-a5d9-134a40ad956c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/902d51a8-2407-4f4b-a5d9-134a40ad956c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/902d51a8-2407-4f4b-a5d9-134a40ad956c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/902d51a8-2407-4f4b-a5d9-134a40ad956c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/902d51a8-2407-4f4b-a5d9-134a40ad956c/volumes/kubernetes.io~projected/kube-api-access-kbwwx:{mountpoint:/var/lib/kubelet/pods/902d51a8-2407-4f4b-a5d9-134a40ad956c/volumes/kubernetes.io~projected/kube-api-access-kbwwx major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91a74a01-d2c7-42bd-9ead-8bdc48398e28/volumes/kubernetes.io~projected/kube-api-access-28zhc:{mountpoint:/var/lib/kubelet/pods/91a74a01-d2c7-42bd-9ead-8bdc48398e28/volumes/kubernetes.io~projected/kube-api-access-28zhc major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91a74a01-d2c7-42bd-9ead-8bdc48398e28/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/91a74a01-d2c7-42bd-9ead-8bdc48398e28/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4/volumes/kubernetes.io~projected/kube-api-access-77lwg:{mountpoint:/var/lib/kubelet/pods/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4/volumes/kubernetes.io~projected/kube-api-access-77lwg major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f798cfa-1160-4da5-859c-47d81c07f969/volumes/kubernetes.io~projected/kube-api-access-f9cm9:{mountpoint:/var/lib/kubelet/pods/9f798cfa-1160-4da5-859c-47d81c07f969/volumes/kubernetes.io~projected/kube-api-access-f9cm9 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a39f3ef9-2d60-4da6-b69a-9bdfac039833/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/a39f3ef9-2d60-4da6-b69a-9bdfac039833/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a39f3ef9-2d60-4da6-b69a-9bdfac039833/volumes/kubernetes.io~projected/kube-api-access-st4j9:{mountpoint:/var/lib/kubelet/pods/a39f3ef9-2d60-4da6-b69a-9bdfac039833/volumes/kubernetes.io~projected/kube-api-access-st4j9 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a39f3ef9-2d60-4da6-b69a-9bdfac039833/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/a39f3ef9-2d60-4da6-b69a-9bdfac039833/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adb8a888-520e-4ef5-819a-ee9331b0ccd9/volumes/kubernetes.io~projected/kube-api-access-66fbw:{mountpoint:/var/lib/kubelet/pods/adb8a888-520e-4ef5-819a-ee9331b0ccd9/volumes/kubernetes.io~projected/kube-api-access-66fbw major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b88b4dfb-1523-4701-9388-781cedab823b/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/b88b4dfb-1523-4701-9388-781cedab823b/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:366 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c333281d-bd01-4c24-bb15-fa3ee2583fb9/volumes/kubernetes.io~projected/kube-api-access-m8hv9:{mountpoint:/var/lib/kubelet/pods/c333281d-bd01-4c24-bb15-fa3ee2583fb9/volumes/kubernetes.io~projected/kube-api-access-m8hv9 major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c333281d-bd01-4c24-bb15-fa3ee2583fb9/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/c333281d-bd01-4c24-bb15-fa3ee2583fb9/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c333281d-bd01-4c24-bb15-fa3ee2583fb9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/c333281d-bd01-4c24-bb15-fa3ee2583fb9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c333281d-bd01-4c24-bb15-fa3ee2583fb9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/c333281d-bd01-4c24-bb15-fa3ee2583fb9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c333281d-bd01-4c24-bb15-fa3ee2583fb9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/c333281d-bd01-4c24-bb15-fa3ee2583fb9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c333281d-bd01-4c24-bb15-fa3ee2583fb9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/c333281d-bd01-4c24-bb15-fa3ee2583fb9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c333281d-bd01-4c24-bb15-fa3ee2583fb9/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/c333281d-bd01-4c24-bb15-fa3ee2583fb9/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9dd01bc-7209-4fe5-ac79-8efaa6c14232/volumes/kubernetes.io~projected/kube-api-access-qx2wj:{mountpoint:/var/lib/kubelet/pods/e9dd01bc-7209-4fe5-ac79-8efaa6c14232/volumes/kubernetes.io~projected/kube-api-access-qx2wj major:0 minor:388 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9dd01bc-7209-4fe5-ac79-8efaa6c14232/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/e9dd01bc-7209-4fe5-ac79-8efaa6c14232/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:387 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f857cdde-184c-436f-a8bd-3de120963268/volumes/kubernetes.io~projected/kube-api-access-26fmn:{mountpoint:/var/lib/kubelet/pods/f857cdde-184c-436f-a8bd-3de120963268/volumes/kubernetes.io~projected/kube-api-access-26fmn major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f857cdde-184c-436f-a8bd-3de120963268/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/f857cdde-184c-436f-a8bd-3de120963268/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f86263ab-30e1-40bf-a671-f82f44423125/volumes/kubernetes.io~projected/kube-api-access-4zkz5:{mountpoint:/var/lib/kubelet/pods/f86263ab-30e1-40bf-a671-f82f44423125/volumes/kubernetes.io~projected/kube-api-access-4zkz5 major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f86263ab-30e1-40bf-a671-f82f44423125/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/f86263ab-30e1-40bf-a671-f82f44423125/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f86263ab-30e1-40bf-a671-f82f44423125/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/f86263ab-30e1-40bf-a671-f82f44423125/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:276 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/6c96c4fa770cc2b8df2dfd90fb16fe5a90d03018a3048890d2b91f8a418ea54d/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/b84d1c7b6c2fbfdaa5dfcdeb4a22f6d31a7cea6be905dab2df5f4623caf7bd4e/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/4cbd015366040245454c9ae23a7a8585a418f9806e6f42f7d84836ba389c87d6/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/3955b5e3b02f3df815caff9a29616aaf268bc5db5a34dcc10aa817d8e6e32367/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/4020a354e3660a5e509b34979441fdeb6520bbabe0e40bd9235577fce803b9d8/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/51528bb28341746a4a382b348b6c1c39a9c7093c5b75ba8dd6e7aa974bd7807c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/cf329b2c4a053a90e7bba7e1ae4b4eb461c650d150524f5037a2e79b39235371/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/36bcfe973b0fd1967503bc0e2981d230cbd1fbb77c6fb557f81fdf2dd4468f5e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/0e0e045afc66604b625503201dfea26827cee69afbec0dd002450a1563fabe01/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/b22ebc4aa825137575e1e3c4943c04d332c54f9ddcec432d1d841919b4c44691/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/97ab80f5d9062a70e401f033baa4e78d15a53f6471a938a4c0430b774f574a17/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/703435dde28730bb8d36ce5b08afe701209347662682567efdbbc3c33440f075/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a3884a0037ffdbda9cc80da24ef35837af70447a39701c55de68ef02932b2830/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/e8b5959090e05b634cb04bff9309a2ccd47b0ef653eeb496132ad4122943a0dc/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/988f7a40f2e8dc9106badaa9d1330445a720adf968e2c5703f7c5e3dc6a9798a/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/f9c03e2d6769bb056dabea6f0ba5e49e6be8f9fde3f42884dbd08bb1b0c9cfa2/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/d8e10f2660d0553ec4e4cb654c4106aaaffa1c4152acefc9847b0daa7ad835fa/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/3f58ed63211807678a58e0d6e1e0376ac95d0a7253706d5d708fb691557fbf30/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/1e0c0fc9ffeef97c30363a655b4567d61c7f2999f985f3bcf8e3ba0d3a1beceb/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/982d2584e48db1e5c3bedfffd515952e998fa3fa8bddf5fda356721a3b57c84c/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/495f15e6b8ffcf0bf5b802f51ed896bdaa65df00bcee8dab29c20fcd50a93603/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/a27610729eb7f219e5f21b6877bf0b0ed4e7f384c762416814e505a9e10354ed/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/87dfcad8f27695f354d4ca7ac91c4116b7f6822916f76b1b64cad87625d2fc82/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/797b1cd7beee49ce9fdf3b4d3faf79d52a7558fc5e2eb0fc2af660a61ee507ba/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/973c63823dd6e8bc64ca7c8ab927b11883f8a391d866339a92ba8256cfcf083f/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/2d2973aaa4eecef3f57069a05cd15d46405d99aa2c78dcb2dcd82db448761921/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/0bcdcaffdf8cae3b32e3a489573e14f184cd3d6bff9454d2b2905a82ac16dc3a/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/e611e7b41af3a8f0d52b45ff7bc87e82716c6c0e8bf2993b646feb672e4234fd/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/3178004f46feb589da9730d214a2a1357f30d9902a0a2c6d8b107740485de746/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/01b6edd76f6437c3f6c94b5b56047a0a1f75ec5714736409c1111a703195adc7/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/ba0522a764f78ae80df64a45cf2df28aa19b790d58922f2fbb57b0653438c7c7/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/36e12656691583fc1c0f793aa159fbb93f855be066298d00772d4a47f1d2788c/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/1231d4937b01e27aa6a31bc5038c9e9db4e70a1a2c9b4077cf439e7b13acc99f/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/465abcfcbfe858f1fa7ce7082950b55411081556753401956d4e0616da12a34f/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/fc41ac998f5c2654e7567f7c37d4af53da4d431f8c78015d2e27459622a18d5c/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/f72a2145d7577cde5f74886ae7f8404f2c0cea704a347a4b1211cc56897ffa30/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/bce23ebc312ff4f08336b9301b468ec7744c6d045c5ec06648a7fa85aab97423/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/81aed7f3f39ede52549f70f78298fdc6b7bcf2504268d4318dc45b928703193f/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/5fccdd167fba072922ba75b4b6707e5e8eacf28a7ecfa9371d6a15bba273c619/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/02612d1bafcbf21d34632a0f9e2249f01b879761d3758372887455f6b02cebba/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/7cee96c905b88d3ab577f1dc23fc4c05b9c3dce0bac4fe7c73a8702fa16cfb1b/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/8b6a20d256511a1448017363f3423123fd42bec5c69c292ce19b94619b49af28/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/b60046404cfbfd4a85b8db5e0b0e31c08872f94a3efb3015ee4179114eb1a5e4/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/c6e063bc8fa8fc442a329e7e5f1fee5dd45d229b9fe97e87a62944f203084145/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/b4c38fd75f1ddc1a5251e8e9a66cb00ac5b968a7d7bb2a0e7f32bcf7819e98c8/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/cd8e455699098c13f3b55e30e620992d76f96f2f3c3726ba9b6ca9f9b2ad9795/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/165f7a621b4113dfe0835503447dfd6c9af99f21bfc910d7d80df810ae5d7b30/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/9a8e1ef2c5826dc1c0e42269fa04e52a5dbc96599ca683dd1268112a08b454f8/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/1c43d9a37baaf0db90fc03c61caf3d7b4b42d903fe987f026e3ea3e3d9ab145f/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/3b3ac4bcb51be5aa2fedf08ab871aa1e200597aef0ba068729e3301ceb8333f6/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/f19f885c4a8a72f1c73a138dd529de83ca6e2e5b4297831b5a44d13928f970ea/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/45c84bd6012e206aad4a5bdacb0108b812d7256b243b4dee178e9eea0ce62d77/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/5323bd0c4239747a0e066a53a37a52592960a02cc3974a20553449133df14fb5/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/9a57388e3200d3868ccca08be6d05b696632b29db5e35fe3838e4ecc632db3ad/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/f41071f0576ec39afafeb04f6c502ffb5c7cdb8bea608008c48a6aaa7329fd6c/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-403:{mountpoint:/var/lib/containers/storage/overlay/7a129504331d57aaa2851c791f16f54a917a388c40bb4d794c938e28eb2e7075/merged major:0 minor:403 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/f44982b34e78c42debcae2cac047bb0bbadd4b6b3ad6888daa44f8804c4faacb/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/44ef1297099d8feb5f73097e16b20dde5d402d76048061100b37238857767426/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-420:{mountpoint:/var/lib/containers/storage/overlay/a07a9c3a1449faffaeef701e13756cb8a722e59eb45e3a28efe1096069b735cb/merged major:0 minor:420 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/572e4f0b53f32d4b82a047ca6aa4c556ce4ea0992fa14a67c2da6af0f71309d9/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/7c7c05c05ae72393900dac1bd5d2879366e3358661dd148d55e5d97d74f9507f/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/94c9ea064bb86bfafdeb882da3d0f7be961b3c9fb39000addc56ecd387188f0c/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-440:{mountpoint:/var/lib/containers/storage/overlay/efdbe74d3cde117c275d487555d844b855d925a9818779d503e2c609db2de6e8/merged major:0 minor:440 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/f50668f1c733b0a46e18703c5098d3bdb16780e60953c2934ff6d6bffde7f0f3/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-449:{mountpoint:/var/lib/containers/storage/overlay/3e2e282e0d80bcf1dff5ed941e15fb002ccb4edba9d69c8f545fdb7e5a7aeb07/merged major:0 minor:449 fsType:overlay blockSize:0} overlay_0-457:{mountpoint:/var/lib/containers/storage/overlay/33f0b39fb393af7de57e33cb28077b0586b3f8ccacc271a3d93352c764824645/merged major:0 minor:457 fsType:overlay blockSize:0} overlay_0-462:{mountpoint:/var/lib/containers/storage/overlay/eb10813a192eaee5dff5a295da00bed9df60e436b1b79139afc86176e3a6a1c2/merged major:0 minor:462 fsType:overlay blockSize:0} overlay_0-468:{mountpoint:/var/lib/containers/storage/overlay/f6788085d0fe14d37fee11cf2790dbc38bfbb3edb1cd355300b877d9b9b28989/merged major:0 minor:468 fsType:overlay blockSize:0} overlay_0-473:{mountpoint:/var/lib/containers/storage/overlay/afd83d1e28fdddb32513a2f41e54dd4ad9ff3c606745d19dabb11333df6df159/merged major:0 minor:473 fsType:overlay blockSize:0} overlay_0-493:{mountpoint:/var/lib/containers/storage/overlay/e30fdcd67f01c767a8ff30828711144dd8a3ccb02ee298309647e21e078dd77a/merged major:0 minor:493 fsType:overlay blockSize:0} overlay_0-500:{mountpoint:/var/lib/containers/storage/overlay/073d814b832d854c71e9663caba68db1960cc373ec4627e826976437ca0763de/merged major:0 minor:500 fsType:overlay blockSize:0} overlay_0-502:{mountpoint:/var/lib/containers/storage/overlay/b95b5a4287b4766631e74626cb459b22ab629c9004c3e12d3c6bfe087046c8ca/merged major:0 minor:502 fsType:overlay blockSize:0} overlay_0-504:{mountpoint:/var/lib/containers/storage/overlay/7134c38e4a7ce25b575f471242b568916f8a49caddf5224da0593ff35c26e076/merged major:0 minor:504 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/45a595b2e452062e9cafe868283177e24822ee4757cc51a46d168994317cdb74/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/f240eb11b098c2ef8901bf39912415f1b72a270c23d92bb76d36e349a89f2a0d/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/4dba9743814369e262d035f1a3f63aeefca8606fc9d821b1216bc7ef251813b0/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/2a3f8ae08de4d5d9b047c89e8dc2594803f51befae962f5426a0d69f1999c0fd/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/c42d0b14974a0fe5e4f9f4811fdfe5f7e026fee268ca62a94121816823ace5ca/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/f25416df1002029c878b912f2403a0ff017cefcdccddb4b14748883021bb3b30/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/c4aaf13b2f430dcab12773254ba997238f0d7f20c4cb3b0f5a62dbb7830fdfe2/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/8037de87ed8fe947935e500d813253891f679fc7f67229f8e807f1ff1f2f198d/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/2566e3c645b3f898d26526e8ec8022ccb391e36ad02e43b3b90b8fd6ac135507/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/304d074e59029e2cf40bc22100819f41702854ccff585c692c135a10498958f9/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 02 01:32:32.619626 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.619108    5875 manager.go:217] Machine: {Timestamp:2026-11-02 01:32:32.618637902 +0000 UTC m=+0.059674390 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099899 MemoryCapacity:32812023808 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24141869081b12c005544779740014 SystemUUID:ec241418-6908-1b12-c005-544779740014 BootID:86282386-ad68-4619-a26c-eea327b1570b Filesystems:[{Device:/var/lib/kubelet/pods/91a74a01-d2c7-42bd-9ead-8bdc48398e28/volumes/kubernetes.io~projected/kube-api-access-28zhc DeviceMajor:0 DeviceMinor:202 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31a3b3106c6b4575f445dafbec4c249d38b1223bef155785bb0ea2faf6ec9f5c/userdata/shm DeviceMajor:0 DeviceMinor:264 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:312 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/c333281d-bd01-4c24-bb15-fa3ee2583fb9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:339 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:318 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94511ea34e79b2676f7c1c9811db65435d12c7d56f939ce4661f88d3f3ff18ef/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4/volumes/kubernetes.io~projected/kube-api-access-77lwg DeviceMajor:0 DeviceMinor:289 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:491 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:492 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85999952-cf96-48d1-a608-91c84a7bc17c/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:241 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-502 DeviceMajor:0 DeviceMinor:502 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/adb8a888-520e-4ef5-819a-ee9331b0ccd9/volumes/kubernetes.io~projected/kube-api-access-66fbw DeviceMajor:0 DeviceMinor:57 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/69ae9b34a115e17992488ab6bb6a978d66e0bdaf4fdb424715e2eee95957142d/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-462 DeviceMajor:0 DeviceMinor:462 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-500 DeviceMajor:0 DeviceMinor:500 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e02ef8d-7b7b-4de9-a2db-ae32d200a497/volumes/kubernetes.io~projected/kube-api-access-4f4gw DeviceMajor:0 DeviceMinor:53 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/56b9dbaa2578f01ef84a655f37e6ee1c3a1c3602959ecb6b552615b7300c9990/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/792b1392-5f44-47a8-9b86-05ee811046ea/volumes/kubernetes.io~projected/kube-api-access-pkc2j DeviceMajor:0 DeviceMinor:263 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e005dea061ad10babcd2a270d689a9cb0c2b5c78f224d8c1328fd08fda7c2e0/userdata/shm DeviceMajor:0 DeviceMinor:395 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/65f63e43-859a-4d3a-8393-99724782596b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f70370d-3793-4e25-90e8-fc252f5a7596/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:283 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/6a2f0fa5-cbce-4695-aa38-e8bbeb720280/volumes/kubernetes.io~projected/kube-api-access-c9sj2 DeviceMajor:0 DeviceMinor:369 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/9f798cfa-1160-4da5-859c-47d81c07f969/volumes/kubernetes.io~projected/kube-api-access-f9cm9 DeviceMajor:0 DeviceMinor:59 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a39f3ef9-2d60-4da6-b69a-9bdfac039833/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/f857cdde-184c-436f-a8bd-3de120963268/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9dd01bc-7209-4fe5-ac79-8efaa6c14232/volumes/kubernetes.io~projected/kube-api-access-qx2wj DeviceMajor:0 DeviceMinor:388 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/544ee005-9680-4d16-ae14-a2633763bf10/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:390 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-440 DeviceMajor:0 DeviceMinor:440 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:288 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:318 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-449 DeviceMajor:0 DeviceMinor:449 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8181c311e1c1935dc496d928f2500bd28e19de558f63de49bf74791c75f5bb03/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/f857cdde-184c-436f-a8bd-3de120963268/volumes/kubernetes.io~projected/kube-api-access-26fmn DeviceMajor:0 DeviceMinor:203 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:486 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/841f3428-10c6-4079-a256-cf6e5cd21806/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:358 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c333281d-bd01-4c24-bb15-fa3ee2583fb9/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:342 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:317 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:489 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:319 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~projected/kube-api-access-gpswh DeviceMajor:0 DeviceMinor:320 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3a1f4347a710d8d6bf041568b564311aa0529b25267be62fb4e42109c6a0afae/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:318 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:487 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28/volumes/kubernetes.io~projected/kube-api-access-5vmxb DeviceMajor:0 DeviceMinor:52 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:314 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/902d51a8-2407-4f4b-a5d9-134a40ad956c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c333281d-bd01-4c24-bb15-fa3ee2583fb9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:343 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:485 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/f86263ab-30e1-40bf-a671-f82f44423125/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:280 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:318 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-504 DeviceMajor:0 DeviceMinor:504 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a39f3ef9-2d60-4da6-b69a-9bdfac039833/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59/volumes/kubernetes.io~projected/kube-api-access-jt4vp DeviceMajor:0 DeviceMinor:204 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/c333281d-bd01-4c24-bb15-fa3ee2583fb9/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:335 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7dba8084753fc3d3c7fb38ca22890cf10dd689e50a76867c80541fd8ee48baf8/userdata/shm DeviceMajor:0 DeviceMinor:375 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b88b4dfb-1523-4701-9388-781cedab823b/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:366 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406011904 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/902d51a8-2407-4f4b-a5d9-134a40ad956c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e98988375935e3f025b17c237709d84c2173ddc0314bbecd7df3a7c758006e8d/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:287 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:316 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:317 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/212349606df6c854d29a533d0079cf270f94912312bbddc935e656a210c5b596/userdata/shm DeviceMajor:0 DeviceMinor:376 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/173c5a4c499cae3df6ed358eb93376aa21a1fa387ba8ee04b6aad1f8dab9f9cb/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/c333281d-bd01-4c24-bb15-fa3ee2583fb9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:341 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:490 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eebedc5242e32bfe4d2b763cd48f8cedf8ea1af380030a8cdd9f1cfc83993024/userdata/shm DeviceMajor:0 DeviceMinor:495 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-473 DeviceMajor:0 DeviceMinor:473 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~projected/kube-api-access-qc7c6 DeviceMajor:0 DeviceMinor:478 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ea0457b904929e89ca2f6b5468317750d056f07850e062ed00d655a5c8ad218/userdata/shm DeviceMajor:0 DeviceMinor:285 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/544ee005-9680-4d16-ae14-a2633763bf10/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:391 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-468 DeviceMajor:0 DeviceMinor:468 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:488 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-403 DeviceMajor:0 DeviceMinor:403 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-457 DeviceMajor:0 DeviceMinor:457 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a39f3ef9-2d60-4da6-b69a-9bdfac039833/volumes/kubernetes.io~projected/kube-api-access-st4j9 DeviceMajor:0 DeviceMinor:56 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:311 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/069fc7fc54846606f1108d55c7f501f0612a442463133eb380c50e044f2b5eef/userdata/shm DeviceMajor:0 DeviceMinor:370 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9dd01bc-7209-4fe5-ac79-8efaa6c14232/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:387 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1a39fcc2e1764800b057926da3ca002baee535c03f38487c841ddb9afed6adeb/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b93ed80f94e779453d490ac6563b3bc4fe54645d777edb0179458eb585a87bb1/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/792b1392-5f44-47a8-9b86-05ee811046ea/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:257 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/544ee005-9680-4d16-ae14-a2633763bf10/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:389 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:484 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9662347313a235d1d0e72eb393df4285943e8fbfe4cee5d50247970837213a04/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f65a11b3eb88e9616d6d3b20d732fe322d46788e9eb4fea4e0de2f2c967838c5/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:313 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/c333281d-bd01-4c24-bb15-fa3ee2583fb9/volumes/kubernetes.io~projected/kube-api-access-m8hv9 DeviceMajor:0 DeviceMinor:351 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:483 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/841f3428-10c6-4079-a256-cf6e5cd21806/volumes/kubernetes.io~projected/kube-api-access-2nd6c DeviceMajor:0 DeviceMinor:55 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0db0a4440d26849b6911b476f8ddda7e31933f0ea3721836115dae07380692b7/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/c333281d-bd01-4c24-bb15-fa3ee2583fb9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:340 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-493 DeviceMajor:0 DeviceMinor:493 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/74a59b76abefddb3b42ae880ccfb8df7538d03ad03fee714c400b67133f82159/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/91a74a01-d2c7-42bd-9ead-8bdc48398e28/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f70370d-3793-4e25-90e8-fc252f5a7596/volumes/kubernetes.io~projected/kube-api-access-8hl42 DeviceMajor:0 DeviceMinor:284 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/046dd898-c779-4ce2-84ec-dc91afb8323f/volumes/kubernetes.io~projected/kube-api-access-nnbp7 DeviceMajor:0 DeviceMinor:51 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f86263ab-30e1-40bf-a671-f82f44423125/volumes/kubernetes.io~projected/kube-api-access-4zkz5 DeviceMajor:0 DeviceMinor:281 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:315 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6d333f27e662dda79ef0693c4220289ca475279c9948cae04af48d4ce3e68ac/userdata/shm DeviceMajor:0 DeviceMinor:364 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20/volumes/kubernetes.io~projected/kube-api-access-zb7sn DeviceMajor:0 DeviceMinor:58 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/374a8710e7ed9580be3a4b357b8d10b27f6d9121cb57ac4c27d1e0737cdcf3fc/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/792b1392-5f44-47a8-9b86-05ee811046ea/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:262 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/f86263ab-30e1-40bf-a671-f82f44423125/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:276 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/746e119d52d73097b35d0921c93fddceeafd18cdc55e96cb368f9b9a5c54d5be/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f9a4968a094313736ea605c9a4f7a4883f0786b353a37a3e88d8c9fa38ad501/userdata/shm DeviceMajor:0 DeviceMinor:392 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/544ee005-9680-4d16-ae14-a2633763bf10/volumes/kubernetes.io~projected/kube-api-access-rdrcw DeviceMajor:0 DeviceMinor:394 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9ea61f3a326dc5e719210e4d5309b24a1bbcad0df54b4d76cb0258695ad9641/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/5f70370d-3793-4e25-90e8-fc252f5a7596/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:282 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b178ceb687ed740a6de9b4f920ff574b54f851f0634ea61f239e6902bce2af03/userdata/shm DeviceMajor:0 DeviceMinor:321 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b70d62c7a50f16a630ac3255032d18f9af3d701854c30ba24fddf6128ca24204/userdata/shm DeviceMajor:0 DeviceMinor:371 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:482 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/902d51a8-2407-4f4b-a5d9-134a40ad956c/volumes/kubernetes.io~projected/kube-api-access-kbwwx DeviceMajor:0 DeviceMinor:54 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5dbcf879-26f8-4f7b-a071-eafa688b1c05/volumes/kubernetes.io~projected/kube-api-access-7kjnt DeviceMajor:0 DeviceMinor:374 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-420 DeviceMajor:0 DeviceMinor:420 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:483 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:069fc7fc5484660 MacAddress:4a:e9:78:2b:e7:05 Speed:10000 Mtu:8901} {Name:0db0a4440d26849 MacAddress:36:78:71:89:10:08 Speed:10000 Mtu:8901} {Name:212349606df6c85 MacAddress:fa:68:60:81:04:4e Speed:10000 Mtu:8901} {Name:31a3b3106c6b457 MacAddress:6e:7a:de:30:4a:8f Speed:10000 Mtu:8901} {Name:3a1f4347a710d8d MacAddress:de:35:70:75:a0:c2 Speed:10000 Mtu:8901} {Name:7dba8084753fc3d MacAddress:ae:9a:b7:c5:23:90 Speed:10000 Mtu:8901} {Name:7e005dea061ad10 MacAddress:5e:d2:bf:22:4e:d0 Speed:10000 Mtu:8901} {Name:8f9a4968a094313 MacAddress:8e:ec:c5:75:26:9b Speed:10000 Mtu:8901} {Name:9ea0457b904929e MacAddress:5e:d9:a4:82:2a:6d Speed:10000 Mtu:8901} {Name:b178ceb687ed740 MacAddress:1a:5a:f7:76:d8:eb Speed:10000 Mtu:8901} {Name:b6d333f27e662dd MacAddress:ea:1b:67:03:8a:66 Speed:10000 Mtu:8901} {Name:b70d62c7a50f16a MacAddress:9a:dd:bf:18:63:1c Speed:10000 Mtu:8901} {Name:b93ed80f94e7794 MacAddress:c2:04:3f:33:e6:03 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:56:ef:13:53:cf Speed:0 Mtu:9001} {Name:br-int MacAddress:a2:97:09:b1:9e:1d Speed:0 Mtu:8901} {Name:eebedc5242e32bf MacAddress:36:5c:20:bd:8b:3b Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:56:ef:13:53:cf Speed:0 Mtu:9001} {Name:f65a11b3eb88e96 MacAddress:56:04:64:59:9c:11 Speed:10000 Mtu:8901} {Name:f9ea61f3a326dc5 MacAddress:96:c1:37:0d:17:b4 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:5a:a5:9a:ce:23:41 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ee:d5:1f:8c:a3:ce Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812023808 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 01:32:32.619626 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.619597    5875 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 01:32:32.619872 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.619646    5875 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 01:32:32.619944 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.619917    5875 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 01:32:32.620061 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.619944    5875 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-68.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 01:32:32.620104 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.620063    5875 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 01:32:32.620104 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.620071    5875 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 01:32:32.620104 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.620084    5875 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 01:32:32.620104 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.620102    5875 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 01:32:32.620206 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.620188    5875 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 01:32:32.620387 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.620377    5875 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 01:32:32.620489 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.620481    5875 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 01:32:32.620518 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.620514    5875 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 01:32:32.620545 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.620529    5875 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 01:32:32.620545 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.620538    5875 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 01:32:32.620602 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.620546    5875 kubelet.go:405] "Adding apiserver pod source"
Nov 02 01:32:32.620602 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.620554    5875 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 01:32:32.621292 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.621280    5875 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 01:32:32.621340 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.621295    5875 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 01:32:32.621793 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.621778    5875 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 01:32:32.640828 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.640808    5875 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 01:32:32.640828 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.640826    5875 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 01:32:32.640967 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.640935    5875 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 01:32:32.641156 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.641144    5875 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 01:32:32.641185 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.641175    5875 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 01:32:32.641306 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.641297    5875 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 01:32:32.641384 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.641317    5875 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 01:32:32.641384 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.641326    5875 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 01:32:32.641384 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.641332    5875 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 01:32:32.641384 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.641339    5875 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 01:32:32.641384 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.641344    5875 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 01:32:32.641384 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.641350    5875 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 01:32:32.641384 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.641355    5875 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 01:32:32.641384 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.641362    5875 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 01:32:32.641384 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.641367    5875 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 01:32:32.641384 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.641376    5875 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 01:32:32.641676 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.641389    5875 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 01:32:32.641676 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.641419    5875 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 01:32:32.641676 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.641425    5875 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 01:32:32.645056 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.645039    5875 server.go:1263] "Started kubelet"
Nov 02 01:32:32.645154 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.645132    5875 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 01:32:32.645244 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.645163    5875 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 01:32:32.645306 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.645259    5875 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 01:32:32.645670 ip-10-0-141-68 systemd[1]: Started Kubernetes Kubelet.
Nov 02 01:32:32.646952 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.645734    5875 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 01:32:32.646952 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.646461    5875 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 01:32:32.651896 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.651878    5875 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 01:32:32.654127 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.651901    5875 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 01:32:32.654127 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.651939    5875 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 01:26:19 +0000 UTC" deadline="2028-08-04 10:44:47.395615818 +0000 UTC"
Nov 02 01:32:32.654127 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.651957    5875 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15393h12m14.743661771s"
Nov 02 01:32:32.654127 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.651973    5875 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 01:32:32.654127 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.651982    5875 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 01:32:32.654127 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:32:32.651987    5875 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-68.us-west-1.compute.internal\" not found"
Nov 02 01:32:32.654127 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.652133    5875 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 01:32:32.654127 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.654010    5875 factory.go:55] Registering systemd factory
Nov 02 01:32:32.654127 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.654029    5875 factory.go:223] Registration of the systemd container factory successfully
Nov 02 01:32:32.654709 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.654450    5875 factory.go:153] Registering CRI-O factory
Nov 02 01:32:32.654709 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.654466    5875 factory.go:223] Registration of the crio container factory successfully
Nov 02 01:32:32.654709 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.654515    5875 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 01:32:32.654709 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.654533    5875 factory.go:103] Registering Raw factory
Nov 02 01:32:32.654709 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.654547    5875 manager.go:1199] Started watching for new ooms in manager
Nov 02 01:32:32.654929 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.654858    5875 manager.go:319] Starting recovery of all containers
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655124    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32a38f9a-7d31-4a28-8a1b-8653ac3b7a20" volumeName="kubernetes.io/configmap/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-multus-daemon-config" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655183    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="544ee005-9680-4d16-ae14-a2633763bf10" volumeName="kubernetes.io/secret/544ee005-9680-4d16-ae14-a2633763bf10-client-ca-bundle" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655200    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f70370d-3793-4e25-90e8-fc252f5a7596" volumeName="kubernetes.io/configmap/5f70370d-3793-4e25-90e8-fc252f5a7596-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655213    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91a74a01-d2c7-42bd-9ead-8bdc48398e28" volumeName="kubernetes.io/configmap/91a74a01-d2c7-42bd-9ead-8bdc48398e28-config-volume" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655245    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91a74a01-d2c7-42bd-9ead-8bdc48398e28" volumeName="kubernetes.io/empty-dir/91a74a01-d2c7-42bd-9ead-8bdc48398e28-tmp-dir" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655257    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a39f3ef9-2d60-4da6-b69a-9bdfac039833" volumeName="kubernetes.io/secret/a39f3ef9-2d60-4da6-b69a-9bdfac039833-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655271    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c333281d-bd01-4c24-bb15-fa3ee2583fb9" volumeName="kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-tls" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655282    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e0fdbea-232a-4a5e-abe4-3869da804c72" volumeName="kubernetes.io/projected/0e0fdbea-232a-4a5e-abe4-3869da804c72-kube-api-access-gpswh" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655297    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5dbcf879-26f8-4f7b-a071-eafa688b1c05" volumeName="kubernetes.io/projected/5dbcf879-26f8-4f7b-a071-eafa688b1c05-kube-api-access-7kjnt" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655311    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f70370d-3793-4e25-90e8-fc252f5a7596" volumeName="kubernetes.io/configmap/5f70370d-3793-4e25-90e8-fc252f5a7596-metrics-client-ca" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655324    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f70370d-3793-4e25-90e8-fc252f5a7596" volumeName="kubernetes.io/empty-dir/5f70370d-3793-4e25-90e8-fc252f5a7596-volume-directive-shadow" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655335    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65f63e43-859a-4d3a-8393-99724782596b" volumeName="kubernetes.io/configmap/65f63e43-859a-4d3a-8393-99724782596b-konnectivity-ca" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655346    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e02ef8d-7b7b-4de9-a2db-ae32d200a497" volumeName="kubernetes.io/configmap/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-cni-binary-copy" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655358    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="792b1392-5f44-47a8-9b86-05ee811046ea" volumeName="kubernetes.io/configmap/792b1392-5f44-47a8-9b86-05ee811046ea-metrics-client-ca" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655373    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c333281d-bd01-4c24-bb15-fa3ee2583fb9" volumeName="kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655387    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e0fdbea-232a-4a5e-abe4-3869da804c72" volumeName="kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655398    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e0fdbea-232a-4a5e-abe4-3869da804c72" volumeName="kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655410    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31ccc613-71f9-40b5-a1f9-9254f9e7598b" volumeName="kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-thanos-prometheus-http-client-file" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655424    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f70370d-3793-4e25-90e8-fc252f5a7596" volumeName="kubernetes.io/projected/5f70370d-3793-4e25-90e8-fc252f5a7596-kube-api-access-8hl42" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655437    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="841f3428-10c6-4079-a256-cf6e5cd21806" volumeName="kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655448    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91a74a01-d2c7-42bd-9ead-8bdc48398e28" volumeName="kubernetes.io/secret/91a74a01-d2c7-42bd-9ead-8bdc48398e28-metrics-tls" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655462    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bbd7104-ba74-4c31-8cfb-3f307da4b3f4" volumeName="kubernetes.io/configmap/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-metrics-client-ca" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655474    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c333281d-bd01-4c24-bb15-fa3ee2583fb9" volumeName="kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655486    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59" volumeName="kubernetes.io/secret/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655500    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31ccc613-71f9-40b5-a1f9-9254f9e7598b" volumeName="kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655512    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65f63e43-859a-4d3a-8393-99724782596b" volumeName="kubernetes.io/secret/65f63e43-859a-4d3a-8393-99724782596b-agent-certs" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655523    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bbd7104-ba74-4c31-8cfb-3f307da4b3f4" volumeName="kubernetes.io/empty-dir/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-textfile" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655536    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f798cfa-1160-4da5-859c-47d81c07f969" volumeName="kubernetes.io/empty-dir/9f798cfa-1160-4da5-859c-47d81c07f969-tmp-dir" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655552    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c333281d-bd01-4c24-bb15-fa3ee2583fb9" volumeName="kubernetes.io/configmap/c333281d-bd01-4c24-bb15-fa3ee2583fb9-metrics-client-ca" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655564    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9dd01bc-7209-4fe5-ac79-8efaa6c14232" volumeName="kubernetes.io/projected/e9dd01bc-7209-4fe5-ac79-8efaa6c14232-kube-api-access-qx2wj" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655576    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f857cdde-184c-436f-a8bd-3de120963268" volumeName="kubernetes.io/secret/f857cdde-184c-436f-a8bd-3de120963268-cert" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655589    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59" volumeName="kubernetes.io/projected/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-kube-api-access-jt4vp" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655600    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31ccc613-71f9-40b5-a1f9-9254f9e7598b" volumeName="kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655612    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31ccc613-71f9-40b5-a1f9-9254f9e7598b" volumeName="kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-grpc-tls" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655623    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31ccc613-71f9-40b5-a1f9-9254f9e7598b" volumeName="kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655645    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="544ee005-9680-4d16-ae14-a2633763bf10" volumeName="kubernetes.io/projected/544ee005-9680-4d16-ae14-a2633763bf10-kube-api-access-rdrcw" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655659    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="792b1392-5f44-47a8-9b86-05ee811046ea" volumeName="kubernetes.io/projected/792b1392-5f44-47a8-9b86-05ee811046ea-kube-api-access-pkc2j" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655682    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bbd7104-ba74-4c31-8cfb-3f307da4b3f4" volumeName="kubernetes.io/projected/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-kube-api-access-77lwg" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655696    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bbd7104-ba74-4c31-8cfb-3f307da4b3f4" volumeName="kubernetes.io/secret/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-tls" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655708    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e0fdbea-232a-4a5e-abe4-3869da804c72" volumeName="kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655720    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="544ee005-9680-4d16-ae14-a2633763bf10" volumeName="kubernetes.io/secret/544ee005-9680-4d16-ae14-a2633763bf10-secret-metrics-server-tls" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655732    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="841f3428-10c6-4079-a256-cf6e5cd21806" volumeName="kubernetes.io/projected/841f3428-10c6-4079-a256-cf6e5cd21806-kube-api-access-2nd6c" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655745    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a39f3ef9-2d60-4da6-b69a-9bdfac039833" volumeName="kubernetes.io/configmap/a39f3ef9-2d60-4da6-b69a-9bdfac039833-env-overrides" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655757    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a39f3ef9-2d60-4da6-b69a-9bdfac039833" volumeName="kubernetes.io/configmap/a39f3ef9-2d60-4da6-b69a-9bdfac039833-ovnkube-config" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655770    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adb8a888-520e-4ef5-819a-ee9331b0ccd9" volumeName="kubernetes.io/projected/adb8a888-520e-4ef5-819a-ee9331b0ccd9-kube-api-access-66fbw" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655785    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f857cdde-184c-436f-a8bd-3de120963268" volumeName="kubernetes.io/projected/f857cdde-184c-436f-a8bd-3de120963268-kube-api-access-26fmn" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655797    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f86263ab-30e1-40bf-a671-f82f44423125" volumeName="kubernetes.io/secret/f86263ab-30e1-40bf-a671-f82f44423125-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655812    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5dbcf879-26f8-4f7b-a071-eafa688b1c05" volumeName="kubernetes.io/empty-dir/5dbcf879-26f8-4f7b-a071-eafa688b1c05-tmp" seLinuxMountContext=""
Nov 02 01:32:32.655785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655825    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e02ef8d-7b7b-4de9-a2db-ae32d200a497" volumeName="kubernetes.io/configmap/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655837    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bbd7104-ba74-4c31-8cfb-3f307da4b3f4" volumeName="kubernetes.io/configmap/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655850    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c333281d-bd01-4c24-bb15-fa3ee2583fb9" volumeName="kubernetes.io/empty-dir/c333281d-bd01-4c24-bb15-fa3ee2583fb9-tmp" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655862    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c333281d-bd01-4c24-bb15-fa3ee2583fb9" volumeName="kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655874    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f86263ab-30e1-40bf-a671-f82f44423125" volumeName="kubernetes.io/configmap/f86263ab-30e1-40bf-a671-f82f44423125-metrics-client-ca" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655887    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f86263ab-30e1-40bf-a671-f82f44423125" volumeName="kubernetes.io/secret/f86263ab-30e1-40bf-a671-f82f44423125-openshift-state-metrics-tls" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655901    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31ccc613-71f9-40b5-a1f9-9254f9e7598b" volumeName="kubernetes.io/projected/31ccc613-71f9-40b5-a1f9-9254f9e7598b-kube-api-access-qc7c6" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655913    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31ccc613-71f9-40b5-a1f9-9254f9e7598b" volumeName="kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-tls" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655953    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9dd01bc-7209-4fe5-ac79-8efaa6c14232" volumeName="kubernetes.io/secret/e9dd01bc-7209-4fe5-ac79-8efaa6c14232-monitoring-plugin-cert" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655968    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e0fdbea-232a-4a5e-abe4-3869da804c72" volumeName="kubernetes.io/empty-dir/0e0fdbea-232a-4a5e-abe4-3869da804c72-alertmanager-main-db" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655984    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e0fdbea-232a-4a5e-abe4-3869da804c72" volumeName="kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-web-config" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.655996    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31ccc613-71f9-40b5-a1f9-9254f9e7598b" volumeName="kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-web-config" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656020    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32a38f9a-7d31-4a28-8a1b-8653ac3b7a20" volumeName="kubernetes.io/projected/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-kube-api-access-zb7sn" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656052    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902d51a8-2407-4f4b-a5d9-134a40ad956c" volumeName="kubernetes.io/empty-dir/902d51a8-2407-4f4b-a5d9-134a40ad956c-tmp" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656064    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91a74a01-d2c7-42bd-9ead-8bdc48398e28" volumeName="kubernetes.io/projected/91a74a01-d2c7-42bd-9ead-8bdc48398e28-kube-api-access-28zhc" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656129    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a39f3ef9-2d60-4da6-b69a-9bdfac039833" volumeName="kubernetes.io/configmap/a39f3ef9-2d60-4da6-b69a-9bdfac039833-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656162    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a39f3ef9-2d60-4da6-b69a-9bdfac039833" volumeName="kubernetes.io/projected/a39f3ef9-2d60-4da6-b69a-9bdfac039833-kube-api-access-st4j9" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656175    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31ccc613-71f9-40b5-a1f9-9254f9e7598b" volumeName="kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656202    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31ccc613-71f9-40b5-a1f9-9254f9e7598b" volumeName="kubernetes.io/empty-dir/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-db" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656214    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32a38f9a-7d31-4a28-8a1b-8653ac3b7a20" volumeName="kubernetes.io/configmap/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-cni-binary-copy" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656246    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="544ee005-9680-4d16-ae14-a2633763bf10" volumeName="kubernetes.io/configmap/544ee005-9680-4d16-ae14-a2633763bf10-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656270    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="544ee005-9680-4d16-ae14-a2633763bf10" volumeName="kubernetes.io/secret/544ee005-9680-4d16-ae14-a2633763bf10-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656291    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902d51a8-2407-4f4b-a5d9-134a40ad956c" volumeName="kubernetes.io/empty-dir/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-tuned" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656313    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f86263ab-30e1-40bf-a671-f82f44423125" volumeName="kubernetes.io/projected/f86263ab-30e1-40bf-a671-f82f44423125-kube-api-access-4zkz5" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656335    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e0fdbea-232a-4a5e-abe4-3869da804c72" volumeName="kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-config-volume" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656359    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e02ef8d-7b7b-4de9-a2db-ae32d200a497" volumeName="kubernetes.io/projected/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-kube-api-access-4f4gw" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656372    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="792b1392-5f44-47a8-9b86-05ee811046ea" volumeName="kubernetes.io/secret/792b1392-5f44-47a8-9b86-05ee811046ea-prometheus-operator-tls" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656404    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e0fdbea-232a-4a5e-abe4-3869da804c72" volumeName="kubernetes.io/configmap/0e0fdbea-232a-4a5e-abe4-3869da804c72-metrics-client-ca" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656425    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e0fdbea-232a-4a5e-abe4-3869da804c72" volumeName="kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656439    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59" volumeName="kubernetes.io/configmap/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656462    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31ccc613-71f9-40b5-a1f9-9254f9e7598b" volumeName="kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656485    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31ccc613-71f9-40b5-a1f9-9254f9e7598b" volumeName="kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656496    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31ccc613-71f9-40b5-a1f9-9254f9e7598b" volumeName="kubernetes.io/projected/31ccc613-71f9-40b5-a1f9-9254f9e7598b-tls-assets" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656507    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="544ee005-9680-4d16-ae14-a2633763bf10" volumeName="kubernetes.io/empty-dir/544ee005-9680-4d16-ae14-a2633763bf10-audit-log" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656530    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f70370d-3793-4e25-90e8-fc252f5a7596" volumeName="kubernetes.io/secret/5f70370d-3793-4e25-90e8-fc252f5a7596-kube-state-metrics-tls" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656555    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31ccc613-71f9-40b5-a1f9-9254f9e7598b" volumeName="kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656581    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31ccc613-71f9-40b5-a1f9-9254f9e7598b" volumeName="kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-config" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656602    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f70370d-3793-4e25-90e8-fc252f5a7596" volumeName="kubernetes.io/secret/5f70370d-3793-4e25-90e8-fc252f5a7596-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656613    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e02ef8d-7b7b-4de9-a2db-ae32d200a497" volumeName="kubernetes.io/configmap/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656658    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f798cfa-1160-4da5-859c-47d81c07f969" volumeName="kubernetes.io/projected/9f798cfa-1160-4da5-859c-47d81c07f969-kube-api-access-f9cm9" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656680    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adb8a888-520e-4ef5-819a-ee9331b0ccd9" volumeName="kubernetes.io/configmap/adb8a888-520e-4ef5-819a-ee9331b0ccd9-iptables-alerter-script" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656692    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b88b4dfb-1523-4701-9388-781cedab823b" volumeName="kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656714    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="046dd898-c779-4ce2-84ec-dc91afb8323f" volumeName="kubernetes.io/configmap/046dd898-c779-4ce2-84ec-dc91afb8323f-serviceca" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656725    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e0fdbea-232a-4a5e-abe4-3869da804c72" volumeName="kubernetes.io/configmap/0e0fdbea-232a-4a5e-abe4-3869da804c72-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656763    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e0fdbea-232a-4a5e-abe4-3869da804c72" volumeName="kubernetes.io/projected/0e0fdbea-232a-4a5e-abe4-3869da804c72-tls-assets" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656775    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31ccc613-71f9-40b5-a1f9-9254f9e7598b" volumeName="kubernetes.io/empty-dir/31ccc613-71f9-40b5-a1f9-9254f9e7598b-config-out" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656797    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31ccc613-71f9-40b5-a1f9-9254f9e7598b" volumeName="kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-kube-rbac-proxy" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656818    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902d51a8-2407-4f4b-a5d9-134a40ad956c" volumeName="kubernetes.io/projected/902d51a8-2407-4f4b-a5d9-134a40ad956c-kube-api-access-kbwwx" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656841    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bbd7104-ba74-4c31-8cfb-3f307da4b3f4" volumeName="kubernetes.io/secret/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656852    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c333281d-bd01-4c24-bb15-fa3ee2583fb9" volumeName="kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656863    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="046dd898-c779-4ce2-84ec-dc91afb8323f" volumeName="kubernetes.io/projected/046dd898-c779-4ce2-84ec-dc91afb8323f-kube-api-access-nnbp7" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656877    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e0fdbea-232a-4a5e-abe4-3869da804c72" volumeName="kubernetes.io/empty-dir/0e0fdbea-232a-4a5e-abe4-3869da804c72-config-out" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656900    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d19e74b-e519-4c0a-a3aa-956c6b3bcb28" volumeName="kubernetes.io/projected/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-kube-api-access-5vmxb" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656924    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c333281d-bd01-4c24-bb15-fa3ee2583fb9" volumeName="kubernetes.io/projected/c333281d-bd01-4c24-bb15-fa3ee2583fb9-kube-api-access-m8hv9" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.656945    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c333281d-bd01-4c24-bb15-fa3ee2583fb9" volumeName="kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-grpc-tls" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.657007    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e0fdbea-232a-4a5e-abe4-3869da804c72" volumeName="kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-cluster-tls-config" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.657019    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31ccc613-71f9-40b5-a1f9-9254f9e7598b" volumeName="kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-metrics-client-ca" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.657047    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31ccc613-71f9-40b5-a1f9-9254f9e7598b" volumeName="kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.657067    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31ccc613-71f9-40b5-a1f9-9254f9e7598b" volumeName="kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-metrics-client-certs" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.657081    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="544ee005-9680-4d16-ae14-a2633763bf10" volumeName="kubernetes.io/configmap/544ee005-9680-4d16-ae14-a2633763bf10-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.657110    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a2f0fa5-cbce-4695-aa38-e8bbeb720280" volumeName="kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.657127    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="792b1392-5f44-47a8-9b86-05ee811046ea" volumeName="kubernetes.io/secret/792b1392-5f44-47a8-9b86-05ee811046ea-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.657139    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85999952-cf96-48d1-a608-91c84a7bc17c" volumeName="kubernetes.io/secret/85999952-cf96-48d1-a608-91c84a7bc17c-tls-certificates" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.657163    5875 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59" volumeName="kubernetes.io/empty-dir/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-data-volume" seLinuxMountContext=""
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.657174    5875 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 01:32:32.658242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.657182    5875 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 01:32:32.683569 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.683539    5875 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 01:32:32.686165 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.686136    5875 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 01:32:32.686165 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.686170    5875 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 01:32:32.686366 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.686192    5875 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 01:32:32.686366 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:32:32.686254    5875 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 01:32:32.699145 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.699114    5875 generic.go:356] "Generic (PLEG): container finished" podID="efa1d9dffd42c4f4f5c559910849b4c9" containerID="42d7888ebbaf80be1529103f2845105888ef4b5e0f45756620fe6cd5dab8d2a0" exitCode=0
Nov 02 01:32:32.719437 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.719412    5875 generic.go:356] "Generic (PLEG): container finished" podID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerID="3ee78722ab1691cad1750ad559dc90bcb26c8b3c3120835a012d2d0e70cbeeba" exitCode=0
Nov 02 01:32:32.721688 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.721658    5875 generic.go:356] "Generic (PLEG): container finished" podID="9bbd7104-ba74-4c31-8cfb-3f307da4b3f4" containerID="2bef5272c303e612a2d2e915b53534d9872ae79ec6593e02f8ecddd7b6caf3b9" exitCode=0
Nov 02 01:32:32.740902 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.740872    5875 generic.go:356] "Generic (PLEG): container finished" podID="6e02ef8d-7b7b-4de9-a2db-ae32d200a497" containerID="183dc1c619fb4a32b6c6979e3f9c2415ffaf979f6f0da62cb11b43c26314ae35" exitCode=0
Nov 02 01:32:32.740902 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.740899    5875 generic.go:356] "Generic (PLEG): container finished" podID="6e02ef8d-7b7b-4de9-a2db-ae32d200a497" containerID="c2e715d07decb52d5aeeae4af4e7e221f330030008468667498ddd1d21a9cdde" exitCode=0
Nov 02 01:32:32.740902 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.740906    5875 generic.go:356] "Generic (PLEG): container finished" podID="6e02ef8d-7b7b-4de9-a2db-ae32d200a497" containerID="ac49da3ce13eeed309476d58183bb5b3b6287c29fa03dffb5aee884d03028e1f" exitCode=0
Nov 02 01:32:32.740902 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.740911    5875 generic.go:356] "Generic (PLEG): container finished" podID="6e02ef8d-7b7b-4de9-a2db-ae32d200a497" containerID="ceb685ab5c72108b8d0ff2a402a4a6e11dee59ee926b81ce2cadb422d81f2683" exitCode=0
Nov 02 01:32:32.741200 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.740917    5875 generic.go:356] "Generic (PLEG): container finished" podID="6e02ef8d-7b7b-4de9-a2db-ae32d200a497" containerID="72889d8bba7bc213e6beae588070bf9b89130aebab136e743923e9260d993cf5" exitCode=0
Nov 02 01:32:32.741200 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.740922    5875 generic.go:356] "Generic (PLEG): container finished" podID="6e02ef8d-7b7b-4de9-a2db-ae32d200a497" containerID="fae0dde6c84757eadc3faeae582c09d223d142a10ed77600932d32219def7661" exitCode=0
Nov 02 01:32:32.752133 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:32:32.752065    5875 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-68.us-west-1.compute.internal\" not found"
Nov 02 01:32:32.753874 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.753841    5875 generic.go:356] "Generic (PLEG): container finished" podID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerID="21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262" exitCode=0
Nov 02 01:32:32.782578 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.782554    5875 manager.go:324] Recovery completed
Nov 02 01:32:32.786317 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:32:32.786298    5875 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 02 01:32:32.806377 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.806355    5875 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:32:32.808357 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.808332    5875 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:32:32.808433 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.808363    5875 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:32:32.808433 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.808374    5875 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:32:32.811169 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.811148    5875 cpu_manager.go:226] "Starting" policy="none"
Nov 02 01:32:32.811169 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.811166    5875 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 01:32:32.811304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.811187    5875 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 01:32:32.811304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.811169    5875 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:32:32.811393 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.811358    5875 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 01:32:32.811450 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.811375    5875 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 01:32:32.811450 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.811403    5875 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 01:32:32.811450 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.811411    5875 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 01:32:32.811450 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.811417    5875 policy_none.go:50] "Start"
Nov 02 01:32:32.811450 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.811428    5875 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 01:32:32.811450 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.811438    5875 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 01:32:32.811628 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.811572    5875 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 01:32:32.811628 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.811581    5875 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 01:32:32.811628 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.811589    5875 policy_none.go:44] "Start"
Nov 02 01:32:32.812156 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.812139    5875 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:32:32.812246 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.812171    5875 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:32:32.812246 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.812186    5875 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:32:32.816481 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.816457    5875 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 01:32:32.819172 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.819157    5875 manager.go:341] "Starting Device Plugin manager"
Nov 02 01:32:32.819281 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:32:32.819182    5875 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 01:32:32.819281 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.819191    5875 server.go:87] "Starting device plugin registration server"
Nov 02 01:32:32.819519 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.819505    5875 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 01:32:32.819569 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.819520    5875 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 01:32:32.819652 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.819632    5875 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 01:32:32.819924 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.819884    5875 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 01:32:32.820038 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.819932    5875 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 01:32:32.820038 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.819939    5875 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 01:32:32.820538 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.819995    5875 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T01:32:32.819920586Z","UUID":"2dfa395d-0049-4aec-9ca6-da8e9bc95602","Handler":null,"Name":"","Endpoint":""}
Nov 02 01:32:32.821544 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:32:32.821523    5875 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 01:32:32.821693 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:32:32.821563    5875 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-68.us-west-1.compute.internal\" not found"
Nov 02 01:32:32.821693 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.821604    5875 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 01:32:32.821693 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.821638    5875 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 01:32:32.830752 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.830717    5875 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:32:32.831482 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.831461    5875 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:32:32.831599 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.831583    5875 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 01:32:32.831756 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.831735    5875 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:32:32.920330 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.920286    5875 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:32:32.921479 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.921456    5875 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:32:32.921601 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.921493    5875 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:32:32.921601 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.921508    5875 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:32:32.921601 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.921565    5875 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:32:32.928924 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.928910    5875 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:32:32.987338 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.987288    5875 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal"]
Nov 02 01:32:32.987992 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.987948    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal" event={"ID":"efa1d9dffd42c4f4f5c559910849b4c9","Type":"ContainerStarted","Data":"038c8456a8da65c5a13a62eee8ba07498f12e36406f999e41c8498bf2c09f489"}
Nov 02 01:32:32.988086 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.987996    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal" event={"ID":"efa1d9dffd42c4f4f5c559910849b4c9","Type":"ContainerDied","Data":"42d7888ebbaf80be1529103f2845105888ef4b5e0f45756620fe6cd5dab8d2a0"}
Nov 02 01:32:32.988086 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.988004    5875 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:32:32.988086 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.988011    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal" event={"ID":"efa1d9dffd42c4f4f5c559910849b4c9","Type":"ContainerStarted","Data":"94511ea34e79b2676f7c1c9811db65435d12c7d56f939ce4661f88d3f3ff18ef"}
Nov 02 01:32:32.988286 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.988119    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal" event={"ID":"5845b2d4146c451ff8e611722d7c3e34","Type":"ContainerStarted","Data":"5037651ea95ef8240a43a9e97ccdcfd1766652cf2ddac2da59cb0331fa23357b"}
Nov 02 01:32:32.988286 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.988138    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal" event={"ID":"5845b2d4146c451ff8e611722d7c3e34","Type":"ContainerStarted","Data":"374a8710e7ed9580be3a4b357b8d10b27f6d9121cb57ac4c27d1e0737cdcf3fc"}
Nov 02 01:32:32.988286 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:32.988176    5875 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:32:33.030536 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.030466    5875 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:32:33.030633 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.030541    5875 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:32:33.049713 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.049685    5875 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:32:33.049824 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:32:33.049732    5875 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:32:33.050241 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.050203    5875 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:32:33.050280 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:32:33.050267    5875 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:32:33.132685 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.132650    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efa1d9dffd42c4f4f5c559910849b4c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal\" (UID: \"efa1d9dffd42c4f4f5c559910849b4c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:32:33.132685 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.132684    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5845b2d4146c451ff8e611722d7c3e34-config\") pod \"kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal\" (UID: \"5845b2d4146c451ff8e611722d7c3e34\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:32:33.132884 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.132703    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/efa1d9dffd42c4f4f5c559910849b4c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal\" (UID: \"efa1d9dffd42c4f4f5c559910849b4c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:32:33.232899 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.232856    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/efa1d9dffd42c4f4f5c559910849b4c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal\" (UID: \"efa1d9dffd42c4f4f5c559910849b4c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:32:33.232899 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.232895    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efa1d9dffd42c4f4f5c559910849b4c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal\" (UID: \"efa1d9dffd42c4f4f5c559910849b4c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:32:33.233106 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.232911    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5845b2d4146c451ff8e611722d7c3e34-config\") pod \"kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal\" (UID: \"5845b2d4146c451ff8e611722d7c3e34\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:32:33.233106 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.232970    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5845b2d4146c451ff8e611722d7c3e34-config\") pod \"kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal\" (UID: \"5845b2d4146c451ff8e611722d7c3e34\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:32:33.233106 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.232970    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/efa1d9dffd42c4f4f5c559910849b4c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal\" (UID: \"efa1d9dffd42c4f4f5c559910849b4c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:32:33.233106 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.232970    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efa1d9dffd42c4f4f5c559910849b4c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal\" (UID: \"efa1d9dffd42c4f4f5c559910849b4c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:32:33.621028 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.620995    5875 apiserver.go:52] "Watching apiserver"
Nov 02 01:32:33.725509 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.725476    5875 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 01:32:33.726449 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.726418    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-79ffdc6779-dp2ff","kube-system/konnectivity-agent-ft5bg","openshift-dns/dns-default-dbzkb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal","openshift-ovn-kubernetes/ovnkube-node-gfkkb","openshift-insights/insights-runtime-extractor-r4njj","openshift-monitoring/monitoring-plugin-77f4f76594-rc9qm","openshift-multus/multus-additional-cni-plugins-jtl6l","openshift-monitoring/thanos-querier-b84687bcd-x24hf","openshift-multus/multus-7tt8p","openshift-image-registry/node-ca-7xs66","openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c","openshift-multus/network-metrics-daemon-gws4h","openshift-network-diagnostics/network-check-target-gzq27","openshift-console/downloads-6dd789c8b7-lltpm","openshift-network-operator/iptables-alerter-f297t","openshift-monitoring/prometheus-k8s-0","openshift-dns/node-resolver-bd696","kube-system/global-pull-secret-syncer-8vhjc","openshift-cluster-node-tuning-operator/tuned-vh2dn","kube-system/kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal","openshift-ingress-canary/ingress-canary-s275r","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kpwsv","openshift-monitoring/alertmanager-main-0","openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r","openshift-monitoring/node-exporter-rrtpw","openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"]
Nov 02 01:32:33.754527 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.754498    5875 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 01:32:33.768198 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.768170    5875 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:32:33.768348 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.768276    5875 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:32:33.778812 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.778791    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 01:32:33.778900 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.778827    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 01:32:33.779495 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.779478    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 01:32:33.779763 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.779737    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fxpjk\""
Nov 02 01:32:33.779763 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.779743    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-m64v5\""
Nov 02 01:32:33.779927 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.779780    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-42sf2\""
Nov 02 01:32:33.779927 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.779796    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 01:32:33.779927 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.779915    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 01:32:33.780060 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.779943    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 01:32:33.780060 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.779958    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 01:32:33.780060 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.779958    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:32:33.780060 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.779993    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xv9qq\""
Nov 02 01:32:33.780332 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.780317    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 01:32:33.780397 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.780353    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 01:32:33.780397 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.780371    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j52qq\""
Nov 02 01:32:33.780464 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.780395    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 01:32:33.780495 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.780482    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 01:32:33.780495 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.780489    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wfswr\""
Nov 02 01:32:33.780579 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.780562    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 01:32:33.780929 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.780913    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 01:32:33.781085 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.781072    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 01:32:33.781151 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.781144    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 01:32:33.781255 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.781242    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z6qvz\""
Nov 02 01:32:33.781428 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.781416    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vt6x2\""
Nov 02 01:32:33.781485 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.781417    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 01:32:33.781808 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.781795    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 01:32:33.782144 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.782129    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 01:32:33.782196 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.782150    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n6fsn\""
Nov 02 01:32:33.782439 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.782424    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 01:32:33.782439 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.782433    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 01:32:33.782575 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.782561    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 01:32:33.782632 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.782587    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-87db6\""
Nov 02 01:32:33.826120 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.826093    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 01:32:33.826120 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.826119    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 01:32:33.826305 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.826134    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 01:32:33.826352 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.826340    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 02 01:32:33.826513 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.826502    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 01:32:33.826585 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.826570    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 02 01:32:33.826622 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.826608    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 01:32:33.826650 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.826638    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 01:32:33.826650 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.826646    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 01:32:33.828083 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.828069    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 02 01:32:33.828083 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.828081    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 02 01:32:33.828216 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.828101    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 01:32:33.828216 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.828116    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 01:32:33.828216 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.828137    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 01:32:33.830048 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.830030    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 01:32:33.830469 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.830450    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 02 01:32:33.830571 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.830524    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 02 01:32:33.830684 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.830664    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 01:32:33.833645 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.833624    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 02 01:32:33.838171 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838150    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-multus-cni-dir\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.838274 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838187    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28zhc\" (UniqueName: \"kubernetes.io/projected/91a74a01-d2c7-42bd-9ead-8bdc48398e28-kube-api-access-28zhc\") pod \"dns-default-dbzkb\" (UID: \"91a74a01-d2c7-42bd-9ead-8bdc48398e28\") " pod="openshift-dns/dns-default-dbzkb"
Nov 02 01:32:33.838274 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838215    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-systemd-units\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.838398 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838273    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-slash\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.838398 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838296    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-etc-openvswitch\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.838398 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838364    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-cni-binary-copy\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:32:33.838398 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838391    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.838591 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838419    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e0fdbea-232a-4a5e-abe4-3869da804c72-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.838591 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838446    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-cni-netd\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.838591 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838470    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a39f3ef9-2d60-4da6-b69a-9bdfac039833-ovn-node-metrics-cert\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.838591 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838503    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a39f3ef9-2d60-4da6-b69a-9bdfac039833-ovnkube-script-lib\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.838591 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838527    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-data-volume\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:33.838822 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838597    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5dbcf879-26f8-4f7b-a071-eafa688b1c05-tmp\") pod \"downloads-6dd789c8b7-lltpm\" (UID: \"5dbcf879-26f8-4f7b-a071-eafa688b1c05\") " pod="openshift-console/downloads-6dd789c8b7-lltpm"
Nov 02 01:32:33.838822 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838624    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-data-volume\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:33.838822 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838633    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b88b4dfb-1523-4701-9388-781cedab823b-dbus\") pod \"global-pull-secret-syncer-8vhjc\" (UID: \"b88b4dfb-1523-4701-9388-781cedab823b\") " pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:32:33.838822 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838657    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/544ee005-9680-4d16-ae14-a2633763bf10-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:33.838822 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838674    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.838822 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838631    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-cni-binary-copy\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:32:33.838822 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838692    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5dbcf879-26f8-4f7b-a071-eafa688b1c05-tmp\") pod \"downloads-6dd789c8b7-lltpm\" (UID: \"5dbcf879-26f8-4f7b-a071-eafa688b1c05\") " pod="openshift-console/downloads-6dd789c8b7-lltpm"
Nov 02 01:32:33.838822 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838697    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-run-openvswitch\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.838822 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838745    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hl42\" (UniqueName: \"kubernetes.io/projected/5f70370d-3793-4e25-90e8-fc252f5a7596-kube-api-access-8hl42\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:33.838822 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838749    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a39f3ef9-2d60-4da6-b69a-9bdfac039833-ovnkube-script-lib\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.838822 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838774    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.838822 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838800    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-var-lib-kubelet\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.838822 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838815    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e0fdbea-232a-4a5e-abe4-3869da804c72-config-out\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.838822 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838830    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-web-config\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.839329 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838848    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/65f63e43-859a-4d3a-8393-99724782596b-agent-certs\") pod \"konnectivity-agent-ft5bg\" (UID: \"65f63e43-859a-4d3a-8393-99724782596b\") " pod="kube-system/konnectivity-agent-ft5bg"
Nov 02 01:32:33.839329 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838869    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/544ee005-9680-4d16-ae14-a2633763bf10-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:33.839329 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.838921    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e0fdbea-232a-4a5e-abe4-3869da804c72-config-out\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.839329 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839030    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5f70370d-3793-4e25-90e8-fc252f5a7596-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:33.839329 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839062    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/65f63e43-859a-4d3a-8393-99724782596b-agent-certs\") pod \"konnectivity-agent-ft5bg\" (UID: \"65f63e43-859a-4d3a-8393-99724782596b\") " pod="kube-system/konnectivity-agent-ft5bg"
Nov 02 01:32:33.839329 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839062    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-host\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.839329 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839087    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-tuned\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.839329 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839103    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-sys-fs\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:32:33.839329 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839112    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5f70370d-3793-4e25-90e8-fc252f5a7596-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:33.839329 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839061    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.839329 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839118    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-kubernetes\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.839329 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839170    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:33.839329 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839179    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-tuned\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.839329 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839209    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b88b4dfb-1523-4701-9388-781cedab823b-kubelet-config\") pod \"global-pull-secret-syncer-8vhjc\" (UID: \"b88b4dfb-1523-4701-9388-781cedab823b\") " pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:32:33.839329 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839296    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/544ee005-9680-4d16-ae14-a2633763bf10-secret-metrics-server-client-certs\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:33.839986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839381    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.839986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839429    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:32:33.839986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839458    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-device-dir\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:32:33.839986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839485    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-run-multus-certs\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.839986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839512    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66fbw\" (UniqueName: \"kubernetes.io/projected/adb8a888-520e-4ef5-819a-ee9331b0ccd9-kube-api-access-66fbw\") pod \"iptables-alerter-f297t\" (UID: \"adb8a888-520e-4ef5-819a-ee9331b0ccd9\") " pod="openshift-network-operator/iptables-alerter-f297t"
Nov 02 01:32:33.839986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839538    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9cm9\" (UniqueName: \"kubernetes.io/projected/9f798cfa-1160-4da5-859c-47d81c07f969-kube-api-access-f9cm9\") pod \"node-resolver-bd696\" (UID: \"9f798cfa-1160-4da5-859c-47d81c07f969\") " pod="openshift-dns/node-resolver-bd696"
Nov 02 01:32:33.839986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839564    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx2wj\" (UniqueName: \"kubernetes.io/projected/e9dd01bc-7209-4fe5-ac79-8efaa6c14232-kube-api-access-qx2wj\") pod \"monitoring-plugin-77f4f76594-rc9qm\" (UID: \"e9dd01bc-7209-4fe5-ac79-8efaa6c14232\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rc9qm"
Nov 02 01:32:33.839986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839589    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/792b1392-5f44-47a8-9b86-05ee811046ea-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8zqhk\" (UID: \"792b1392-5f44-47a8-9b86-05ee811046ea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk"
Nov 02 01:32:33.839986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839615    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.839986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839644    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77lwg\" (UniqueName: \"kubernetes.io/projected/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-kube-api-access-77lwg\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:33.839986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839684    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kjnt\" (UniqueName: \"kubernetes.io/projected/5dbcf879-26f8-4f7b-a071-eafa688b1c05-kube-api-access-7kjnt\") pod \"downloads-6dd789c8b7-lltpm\" (UID: \"5dbcf879-26f8-4f7b-a071-eafa688b1c05\") " pod="openshift-console/downloads-6dd789c8b7-lltpm"
Nov 02 01:32:33.839986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839720    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbwwx\" (UniqueName: \"kubernetes.io/projected/902d51a8-2407-4f4b-a5d9-134a40ad956c-kube-api-access-kbwwx\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.839986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839741    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/adb8a888-520e-4ef5-819a-ee9331b0ccd9-host-slash\") pod \"iptables-alerter-f297t\" (UID: \"adb8a888-520e-4ef5-819a-ee9331b0ccd9\") " pod="openshift-network-operator/iptables-alerter-f297t"
Nov 02 01:32:33.839986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839757    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/544ee005-9680-4d16-ae14-a2633763bf10-audit-log\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:33.839986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839778    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.839986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839822    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-run-netns\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.839986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839846    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-config-volume\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.839986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839860    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/544ee005-9680-4d16-ae14-a2633763bf10-audit-log\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:33.839986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839869    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-kubelet\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.839986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839899    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-cni-bin\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.839986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839957    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e9dd01bc-7209-4fe5-ac79-8efaa6c14232-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rc9qm\" (UID: \"e9dd01bc-7209-4fe5-ac79-8efaa6c14232\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rc9qm"
Nov 02 01:32:33.839986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.839992    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.840583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840016    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-registration-dir\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:32:33.840583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840036    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-etc-kubernetes\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.840583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840093    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.840583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840121    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/65f63e43-859a-4d3a-8393-99724782596b-konnectivity-ca\") pod \"konnectivity-agent-ft5bg\" (UID: \"65f63e43-859a-4d3a-8393-99724782596b\") " pod="kube-system/konnectivity-agent-ft5bg"
Nov 02 01:32:33.840583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840139    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-log-socket\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.840583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840155    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-sysconfig\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.840583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840172    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:32:33.840583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840211    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9f798cfa-1160-4da5-859c-47d81c07f969-hosts-file\") pod \"node-resolver-bd696\" (UID: \"9f798cfa-1160-4da5-859c-47d81c07f969\") " pod="openshift-dns/node-resolver-bd696"
Nov 02 01:32:33.840583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840257    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.840583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840285    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc7c6\" (UniqueName: \"kubernetes.io/projected/31ccc613-71f9-40b5-a1f9-9254f9e7598b-kube-api-access-qc7c6\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.840583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840314    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/792b1392-5f44-47a8-9b86-05ee811046ea-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8zqhk\" (UID: \"792b1392-5f44-47a8-9b86-05ee811046ea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk"
Nov 02 01:32:33.840583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840331    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/65f63e43-859a-4d3a-8393-99724782596b-konnectivity-ca\") pod \"konnectivity-agent-ft5bg\" (UID: \"65f63e43-859a-4d3a-8393-99724782596b\") " pod="kube-system/konnectivity-agent-ft5bg"
Nov 02 01:32:33.840583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840341    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a39f3ef9-2d60-4da6-b69a-9bdfac039833-env-overrides\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.840583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840374    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-sysctl-conf\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.840583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840392    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:32:33.840583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840403    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdrcw\" (UniqueName: \"kubernetes.io/projected/544ee005-9680-4d16-ae14-a2633763bf10-kube-api-access-rdrcw\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:33.840583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840472    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a39f3ef9-2d60-4da6-b69a-9bdfac039833-env-overrides\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.840583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840502    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-system-cni-dir\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.840583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840543    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e0fdbea-232a-4a5e-abe4-3869da804c72-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.840583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840571    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-run-systemd\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.840583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840590    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st4j9\" (UniqueName: \"kubernetes.io/projected/a39f3ef9-2d60-4da6-b69a-9bdfac039833-kube-api-access-st4j9\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.841304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840642    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f70370d-3793-4e25-90e8-fc252f5a7596-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:33.841304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840665    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.841304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840681    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-var-lib-cni-bin\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.841304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840697    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/91a74a01-d2c7-42bd-9ead-8bdc48398e28-config-volume\") pod \"dns-default-dbzkb\" (UID: \"91a74a01-d2c7-42bd-9ead-8bdc48398e28\") " pod="openshift-dns/dns-default-dbzkb"
Nov 02 01:32:33.841304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840718    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f70370d-3793-4e25-90e8-fc252f5a7596-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:33.841304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840743    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91a74a01-d2c7-42bd-9ead-8bdc48398e28-metrics-tls\") pod \"dns-default-dbzkb\" (UID: \"91a74a01-d2c7-42bd-9ead-8bdc48398e28\") " pod="openshift-dns/dns-default-dbzkb"
Nov 02 01:32:33.841304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840766    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/046dd898-c779-4ce2-84ec-dc91afb8323f-serviceca\") pod \"node-ca-7xs66\" (UID: \"046dd898-c779-4ce2-84ec-dc91afb8323f\") " pod="openshift-image-registry/node-ca-7xs66"
Nov 02 01:32:33.841304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840794    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-sys\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.841304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840819    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:32:33.841304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840846    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret\") pod \"global-pull-secret-syncer-8vhjc\" (UID: \"b88b4dfb-1523-4701-9388-781cedab823b\") " pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:32:33.841304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840859    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f70370d-3793-4e25-90e8-fc252f5a7596-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:33.841304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840882    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-run-k8s-cni-cncf-io\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.841304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841211    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91a74a01-d2c7-42bd-9ead-8bdc48398e28-tmp-dir\") pod \"dns-default-dbzkb\" (UID: \"91a74a01-d2c7-42bd-9ead-8bdc48398e28\") " pod="openshift-dns/dns-default-dbzkb"
Nov 02 01:32:33.841304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840890    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/91a74a01-d2c7-42bd-9ead-8bdc48398e28-config-volume\") pod \"dns-default-dbzkb\" (UID: \"91a74a01-d2c7-42bd-9ead-8bdc48398e28\") " pod="openshift-dns/dns-default-dbzkb"
Nov 02 01:32:33.841304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841140    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:32:33.841304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841255    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.841304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841277    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-tls\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:33.841304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841296    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8hv9\" (UniqueName: \"kubernetes.io/projected/c333281d-bd01-4c24-bb15-fa3ee2583fb9-kube-api-access-m8hv9\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:33.841304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841300    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91a74a01-d2c7-42bd-9ead-8bdc48398e28-tmp-dir\") pod \"dns-default-dbzkb\" (UID: \"91a74a01-d2c7-42bd-9ead-8bdc48398e28\") " pod="openshift-dns/dns-default-dbzkb"
Nov 02 01:32:33.841304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841313    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.841877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841163    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88b4dfb-1523-4701-9388-781cedab823b-original-pull-secret\") pod \"global-pull-secret-syncer-8vhjc\" (UID: \"b88b4dfb-1523-4701-9388-781cedab823b\") " pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:32:33.841877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.840962    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91a74a01-d2c7-42bd-9ead-8bdc48398e28-metrics-tls\") pod \"dns-default-dbzkb\" (UID: \"91a74a01-d2c7-42bd-9ead-8bdc48398e28\") " pod="openshift-dns/dns-default-dbzkb"
Nov 02 01:32:33.841877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841348    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-socket-dir\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:32:33.841877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841365    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/046dd898-c779-4ce2-84ec-dc91afb8323f-serviceca\") pod \"node-ca-7xs66\" (UID: \"046dd898-c779-4ce2-84ec-dc91afb8323f\") " pod="openshift-image-registry/node-ca-7xs66"
Nov 02 01:32:33.841877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841377    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-etc-selinux\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:32:33.841877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841404    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-node-log\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.841877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841442    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a39f3ef9-2d60-4da6-b69a-9bdfac039833-ovnkube-config\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.841877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841492    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-metrics-client-ca\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:33.841877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841503    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.841877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841538    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-run\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.841877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841569    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:32:33.841877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841588    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31ccc613-71f9-40b5-a1f9-9254f9e7598b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.841877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841615    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.841877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841645    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a39f3ef9-2d60-4da6-b69a-9bdfac039833-ovnkube-config\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.841877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841696    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnbp7\" (UniqueName: \"kubernetes.io/projected/046dd898-c779-4ce2-84ec-dc91afb8323f-kube-api-access-nnbp7\") pod \"node-ca-7xs66\" (UID: \"046dd898-c779-4ce2-84ec-dc91afb8323f\") " pod="openshift-image-registry/node-ca-7xs66"
Nov 02 01:32:33.841877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841726    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-run-netns\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.841877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841759    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9sj2\" (UniqueName: \"kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2\") pod \"network-check-target-gzq27\" (UID: \"6a2f0fa5-cbce-4695-aa38-e8bbeb720280\") " pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:32:33.841877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841787    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f798cfa-1160-4da5-859c-47d81c07f969-tmp-dir\") pod \"node-resolver-bd696\" (UID: \"9f798cfa-1160-4da5-859c-47d81c07f969\") " pod="openshift-dns/node-resolver-bd696"
Nov 02 01:32:33.841877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841814    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/544ee005-9680-4d16-ae14-a2633763bf10-metrics-server-audit-profiles\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:33.841877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841841    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31ccc613-71f9-40b5-a1f9-9254f9e7598b-config-out\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.841877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841863    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.842498 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841888    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt4vp\" (UniqueName: \"kubernetes.io/projected/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-kube-api-access-jt4vp\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:33.842498 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841914    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/902d51a8-2407-4f4b-a5d9-134a40ad956c-tmp\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.842498 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841921    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f798cfa-1160-4da5-859c-47d81c07f969-tmp-dir\") pod \"node-resolver-bd696\" (UID: \"9f798cfa-1160-4da5-859c-47d81c07f969\") " pod="openshift-dns/node-resolver-bd696"
Nov 02 01:32:33.842498 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841941    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31ccc613-71f9-40b5-a1f9-9254f9e7598b-config-out\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.842498 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841939    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/85999952-cf96-48d1-a608-91c84a7bc17c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-kpwsv\" (UID: \"85999952-cf96-48d1-a608-91c84a7bc17c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kpwsv"
Nov 02 01:32:33.842498 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.841983    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/544ee005-9680-4d16-ae14-a2633763bf10-secret-metrics-server-tls\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:33.842498 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842006    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/902d51a8-2407-4f4b-a5d9-134a40ad956c-tmp\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.842498 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842009    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0e0fdbea-232a-4a5e-abe4-3869da804c72-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.842498 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842037    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-run-ovn\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.842498 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842060    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-lib-modules\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.842498 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842089    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/544ee005-9680-4d16-ae14-a2633763bf10-metrics-server-audit-profiles\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:33.842498 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842090    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0e0fdbea-232a-4a5e-abe4-3869da804c72-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.842498 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842129    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-multus-socket-dir-parent\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.842498 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842156    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb7sn\" (UniqueName: \"kubernetes.io/projected/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-kube-api-access-zb7sn\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.842498 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842185    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26fmn\" (UniqueName: \"kubernetes.io/projected/f857cdde-184c-436f-a8bd-3de120963268-kube-api-access-26fmn\") pod \"ingress-canary-s275r\" (UID: \"f857cdde-184c-436f-a8bd-3de120963268\") " pod="openshift-ingress-canary/ingress-canary-s275r"
Nov 02 01:32:33.842498 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842259    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nd6c\" (UniqueName: \"kubernetes.io/projected/841f3428-10c6-4079-a256-cf6e5cd21806-kube-api-access-2nd6c\") pod \"network-metrics-daemon-gws4h\" (UID: \"841f3428-10c6-4079-a256-cf6e5cd21806\") " pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:32:33.842498 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842333    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/046dd898-c779-4ce2-84ec-dc91afb8323f-host\") pod \"node-ca-7xs66\" (UID: \"046dd898-c779-4ce2-84ec-dc91afb8323f\") " pod="openshift-image-registry/node-ca-7xs66"
Nov 02 01:32:33.842498 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842456    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:33.842498 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842501    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-modprobe-d\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842522    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-multus-daemon-config\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842563    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs\") pod \"network-metrics-daemon-gws4h\" (UID: \"841f3428-10c6-4079-a256-cf6e5cd21806\") " pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842635    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-textfile\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842665    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f86263ab-30e1-40bf-a671-f82f44423125-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-4snk7\" (UID: \"f86263ab-30e1-40bf-a671-f82f44423125\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842689    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842692    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c333281d-bd01-4c24-bb15-fa3ee2583fb9-metrics-client-ca\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842744    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842752    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-multus-daemon-config\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842758    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-textfile\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842769    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/841f3428-10c6-4079-a256-cf6e5cd21806-metrics-certs\") pod \"network-metrics-daemon-gws4h\" (UID: \"841f3428-10c6-4079-a256-cf6e5cd21806\") " pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842773    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842812    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842832    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5f70370d-3793-4e25-90e8-fc252f5a7596-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842851    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/adb8a888-520e-4ef5-819a-ee9331b0ccd9-iptables-alerter-script\") pod \"iptables-alerter-f297t\" (UID: \"adb8a888-520e-4ef5-819a-ee9331b0ccd9\") " pod="openshift-network-operator/iptables-alerter-f297t"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842872    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-var-lib-openvswitch\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842976    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-run-ovn-kubernetes\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842981    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.842996    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f70370d-3793-4e25-90e8-fc252f5a7596-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843013    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-system-cni-dir\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843060    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f4gw\" (UniqueName: \"kubernetes.io/projected/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-kube-api-access-4f4gw\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843069    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5f70370d-3793-4e25-90e8-fc252f5a7596-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843096    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/544ee005-9680-4d16-ae14-a2633763bf10-client-ca-bundle\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843117    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/adb8a888-520e-4ef5-819a-ee9331b0ccd9-iptables-alerter-script\") pod \"iptables-alerter-f297t\" (UID: \"adb8a888-520e-4ef5-819a-ee9331b0ccd9\") " pod="openshift-network-operator/iptables-alerter-f297t"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843140    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-config\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843158    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843183    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-var-lib-cni-multus\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843208    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-hostroot\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843245    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843268    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpswh\" (UniqueName: \"kubernetes.io/projected/0e0fdbea-232a-4a5e-abe4-3869da804c72-kube-api-access-gpswh\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.843271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843290    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zkz5\" (UniqueName: \"kubernetes.io/projected/f86263ab-30e1-40bf-a671-f82f44423125-kube-api-access-4zkz5\") pod \"openshift-state-metrics-7bd7489957-4snk7\" (UID: \"f86263ab-30e1-40bf-a671-f82f44423125\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843307    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843326    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843340    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-root\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843376    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-tls\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843407    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843433    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-sysctl-d\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843461    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843494    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843536    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-cnibin\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843575    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f86263ab-30e1-40bf-a671-f82f44423125-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-4snk7\" (UID: \"f86263ab-30e1-40bf-a671-f82f44423125\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843606    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-grpc-tls\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843641    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843651    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmxb\" (UniqueName: \"kubernetes.io/projected/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-kube-api-access-5vmxb\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843682    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-cnibin\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843710    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843737    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e0fdbea-232a-4a5e-abe4-3869da804c72-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843761    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-crio-socket\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843785    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-var-lib-kubelet\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843829    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843855    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/792b1392-5f44-47a8-9b86-05ee811046ea-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8zqhk\" (UID: \"792b1392-5f44-47a8-9b86-05ee811046ea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843874    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkc2j\" (UniqueName: \"kubernetes.io/projected/792b1392-5f44-47a8-9b86-05ee811046ea-kube-api-access-pkc2j\") pod \"prometheus-operator-9cd89b74c-8zqhk\" (UID: \"792b1392-5f44-47a8-9b86-05ee811046ea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843899    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-sys\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843916    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-wtmp\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843930    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-systemd\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.843975    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f86263ab-30e1-40bf-a671-f82f44423125-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-4snk7\" (UID: \"f86263ab-30e1-40bf-a671-f82f44423125\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.844027    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.844049    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/792b1392-5f44-47a8-9b86-05ee811046ea-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8zqhk\" (UID: \"792b1392-5f44-47a8-9b86-05ee811046ea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.844057    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-cni-binary-copy\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.844086    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-multus-conf-dir\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.844113    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f857cdde-184c-436f-a8bd-3de120963268-cert\") pod \"ingress-canary-s275r\" (UID: \"f857cdde-184c-436f-a8bd-3de120963268\") " pod="openshift-ingress-canary/ingress-canary-s275r"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.844156    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c333281d-bd01-4c24-bb15-fa3ee2583fb9-tmp\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.844234    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-cni-binary-copy\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.844258    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.844284    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-web-config\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.844478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.844307    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c333281d-bd01-4c24-bb15-fa3ee2583fb9-tmp\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:33.876614 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.876486    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 02 01:32:33.877503 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.877484    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-grbp4\""
Nov 02 01:32:33.877564 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.877484    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 02 01:32:33.877737 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.877712    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 02 01:32:33.877737 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.877722    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 02 01:32:33.877963 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.877765    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-39jc004slgm3b\""
Nov 02 01:32:33.877963 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.877781    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 02 01:32:33.877963 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.877771    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 02 01:32:33.877963 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.877802    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 02 01:32:33.877963 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.877802    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-22x2k\""
Nov 02 01:32:33.877963 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.877824    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 02 01:32:33.877963 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.877902    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 02 01:32:33.877963 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.877911    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 02 01:32:33.877963 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.877942    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 02 01:32:33.877963 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.877922    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 02 01:32:33.877963 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.877957    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 02 01:32:33.878432 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.877945    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-swgv6\""
Nov 02 01:32:33.878432 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.878163    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 02 01:32:33.878432 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.878185    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 02 01:32:33.878432 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.878197    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 02 01:32:33.878432 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.878210    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 02 01:32:33.878432 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.878201    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 02 01:32:33.879115 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.879100    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-xthzw\""
Nov 02 01:32:33.879206 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.879189    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-thfsj\""
Nov 02 01:32:33.879206 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.879196    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.879363 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.879255    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e0fdbea-232a-4a5e-abe4-3869da804c72-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.879363 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.879299    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8tdqx\""
Nov 02 01:32:33.879422 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.879403    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:33.879541 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.879529    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/544ee005-9680-4d16-ae14-a2633763bf10-secret-metrics-server-client-certs\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:33.879759 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.879743    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 02 01:32:33.879878 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.879785    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.879994    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kk5lr\""
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880012    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880054    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880078    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880090    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880092    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6vdkx\""
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880126    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-config-volume\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880141    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880171    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880193    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880201    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8kxtm\""
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880241    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-es4kmf43m5t2c\""
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880251    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880286    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880323    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-bf747\""
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880360    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880382    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880386    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880451    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880470    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-lv26r\""
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880530    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880531    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880530    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/792b1392-5f44-47a8-9b86-05ee811046ea-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8zqhk\" (UID: \"792b1392-5f44-47a8-9b86-05ee811046ea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk"
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880551    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880584    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880608    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880618    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880629    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 02 01:32:33.880674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880661    5875 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:32:33.880718    5875 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880742    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xbzw4\""
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880746    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880756    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qn9km\""
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880793    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880840    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xjcn9\""
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880901    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880953    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880983    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f70370d-3793-4e25-90e8-fc252f5a7596-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.881016    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.881047    5875 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.880724    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.881061    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:32:33.881088    5875 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal"
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.881136    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.881166    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.881140    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.881200    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6csd8\""
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.881397    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3rm6nv61eub59\""
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.881518    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qcsql\""
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.881535    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e0fdbea-232a-4a5e-abe4-3869da804c72-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.881541    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-tls\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.881574    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.881684    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-metrics-client-ca\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.881689    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.881790    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31ccc613-71f9-40b5-a1f9-9254f9e7598b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.882201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.881857    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.883159 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.882875    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/544ee005-9680-4d16-ae14-a2633763bf10-secret-metrics-server-tls\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:33.883159 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.882893    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.883159 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.882904    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/85999952-cf96-48d1-a608-91c84a7bc17c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-kpwsv\" (UID: \"85999952-cf96-48d1-a608-91c84a7bc17c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kpwsv"
Nov 02 01:32:33.883159 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.882914    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f86263ab-30e1-40bf-a671-f82f44423125-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-4snk7\" (UID: \"f86263ab-30e1-40bf-a671-f82f44423125\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"
Nov 02 01:32:33.883159 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.882895    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c333281d-bd01-4c24-bb15-fa3ee2583fb9-metrics-client-ca\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:33.883159 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.883056    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.883159 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.883074    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.884603 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.884321    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.884603 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.884334    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e0fdbea-232a-4a5e-abe4-3869da804c72-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.884603 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.884352    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.884603 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.884363    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/544ee005-9680-4d16-ae14-a2633763bf10-client-ca-bundle\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:33.884603 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.884398    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-grpc-tls\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:33.884603 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.884400    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f86263ab-30e1-40bf-a671-f82f44423125-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-4snk7\" (UID: \"f86263ab-30e1-40bf-a671-f82f44423125\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"
Nov 02 01:32:33.884603 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.884524    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f70370d-3793-4e25-90e8-fc252f5a7596-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:33.884603 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.884533    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f86263ab-30e1-40bf-a671-f82f44423125-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-4snk7\" (UID: \"f86263ab-30e1-40bf-a671-f82f44423125\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"
Nov 02 01:32:33.884603 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.884545    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:33.884603 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.884545    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.884603 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.884565    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.885005 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.884679    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-web-config\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.885005 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.884693    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-tls\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:33.885005 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.884679    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:33.885005 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.884718    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.885005 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.884779    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f857cdde-184c-436f-a8bd-3de120963268-cert\") pod \"ingress-canary-s275r\" (UID: \"f857cdde-184c-436f-a8bd-3de120963268\") " pod="openshift-ingress-canary/ingress-canary-s275r"
Nov 02 01:32:33.885005 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.884799    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c333281d-bd01-4c24-bb15-fa3ee2583fb9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:33.885005 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.884804    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:33.885005 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.884835    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-config\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.889029 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.889005    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a39f3ef9-2d60-4da6-b69a-9bdfac039833-ovn-node-metrics-cert\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.889101 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.889038    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.889193 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.889177    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-web-config\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.890112 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.890091    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/792b1392-5f44-47a8-9b86-05ee811046ea-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8zqhk\" (UID: \"792b1392-5f44-47a8-9b86-05ee811046ea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk"
Nov 02 01:32:33.890210 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.890168    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.890210 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.890192    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.890300 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.890254    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e9dd01bc-7209-4fe5-ac79-8efaa6c14232-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rc9qm\" (UID: \"e9dd01bc-7209-4fe5-ac79-8efaa6c14232\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rc9qm"
Nov 02 01:32:33.896862 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.896844    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28zhc\" (UniqueName: \"kubernetes.io/projected/91a74a01-d2c7-42bd-9ead-8bdc48398e28-kube-api-access-28zhc\") pod \"dns-default-dbzkb\" (UID: \"91a74a01-d2c7-42bd-9ead-8bdc48398e28\") " pod="openshift-dns/dns-default-dbzkb"
Nov 02 01:32:33.897037 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.897021    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hl42\" (UniqueName: \"kubernetes.io/projected/5f70370d-3793-4e25-90e8-fc252f5a7596-kube-api-access-8hl42\") pod \"kube-state-metrics-5dc8d7d89b-vhh4r\" (UID: \"5f70370d-3793-4e25-90e8-fc252f5a7596\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vhh4r"
Nov 02 01:32:33.898609 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.898591    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9cm9\" (UniqueName: \"kubernetes.io/projected/9f798cfa-1160-4da5-859c-47d81c07f969-kube-api-access-f9cm9\") pod \"node-resolver-bd696\" (UID: \"9f798cfa-1160-4da5-859c-47d81c07f969\") " pod="openshift-dns/node-resolver-bd696"
Nov 02 01:32:33.899420 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.899395    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbwwx\" (UniqueName: \"kubernetes.io/projected/902d51a8-2407-4f4b-a5d9-134a40ad956c-kube-api-access-kbwwx\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.899493 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.899417    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77lwg\" (UniqueName: \"kubernetes.io/projected/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-kube-api-access-77lwg\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:33.899631 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.899610    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdrcw\" (UniqueName: \"kubernetes.io/projected/544ee005-9680-4d16-ae14-a2633763bf10-kube-api-access-rdrcw\") pod \"metrics-server-79ffdc6779-dp2ff\" (UID: \"544ee005-9680-4d16-ae14-a2633763bf10\") " pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:33.899706 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.899688    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kjnt\" (UniqueName: \"kubernetes.io/projected/5dbcf879-26f8-4f7b-a071-eafa688b1c05-kube-api-access-7kjnt\") pod \"downloads-6dd789c8b7-lltpm\" (UID: \"5dbcf879-26f8-4f7b-a071-eafa688b1c05\") " pod="openshift-console/downloads-6dd789c8b7-lltpm"
Nov 02 01:32:33.899765 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.899737    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66fbw\" (UniqueName: \"kubernetes.io/projected/adb8a888-520e-4ef5-819a-ee9331b0ccd9-kube-api-access-66fbw\") pod \"iptables-alerter-f297t\" (UID: \"adb8a888-520e-4ef5-819a-ee9331b0ccd9\") " pod="openshift-network-operator/iptables-alerter-f297t"
Nov 02 01:32:33.899898 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.899884    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx2wj\" (UniqueName: \"kubernetes.io/projected/e9dd01bc-7209-4fe5-ac79-8efaa6c14232-kube-api-access-qx2wj\") pod \"monitoring-plugin-77f4f76594-rc9qm\" (UID: \"e9dd01bc-7209-4fe5-ac79-8efaa6c14232\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rc9qm"
Nov 02 01:32:33.901255 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.901205    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8hv9\" (UniqueName: \"kubernetes.io/projected/c333281d-bd01-4c24-bb15-fa3ee2583fb9-kube-api-access-m8hv9\") pod \"thanos-querier-b84687bcd-x24hf\" (UID: \"c333281d-bd01-4c24-bb15-fa3ee2583fb9\") " pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:33.901333 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.901317    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc7c6\" (UniqueName: \"kubernetes.io/projected/31ccc613-71f9-40b5-a1f9-9254f9e7598b-kube-api-access-qc7c6\") pod \"prometheus-k8s-0\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:33.901920 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.901896    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nd6c\" (UniqueName: \"kubernetes.io/projected/841f3428-10c6-4079-a256-cf6e5cd21806-kube-api-access-2nd6c\") pod \"network-metrics-daemon-gws4h\" (UID: \"841f3428-10c6-4079-a256-cf6e5cd21806\") " pod="openshift-multus/network-metrics-daemon-gws4h"
Nov 02 01:32:33.902154 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.902137    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt4vp\" (UniqueName: \"kubernetes.io/projected/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-kube-api-access-jt4vp\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:33.902237 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.902167    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st4j9\" (UniqueName: \"kubernetes.io/projected/a39f3ef9-2d60-4da6-b69a-9bdfac039833-kube-api-access-st4j9\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.902237 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.902184    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnbp7\" (UniqueName: \"kubernetes.io/projected/046dd898-c779-4ce2-84ec-dc91afb8323f-kube-api-access-nnbp7\") pod \"node-ca-7xs66\" (UID: \"046dd898-c779-4ce2-84ec-dc91afb8323f\") " pod="openshift-image-registry/node-ca-7xs66"
Nov 02 01:32:33.902237 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.902186    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb7sn\" (UniqueName: \"kubernetes.io/projected/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-kube-api-access-zb7sn\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.903838 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.903820    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26fmn\" (UniqueName: \"kubernetes.io/projected/f857cdde-184c-436f-a8bd-3de120963268-kube-api-access-26fmn\") pod \"ingress-canary-s275r\" (UID: \"f857cdde-184c-436f-a8bd-3de120963268\") " pod="openshift-ingress-canary/ingress-canary-s275r"
Nov 02 01:32:33.904308 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.904288    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9sj2\" (UniqueName: \"kubernetes.io/projected/6a2f0fa5-cbce-4695-aa38-e8bbeb720280-kube-api-access-c9sj2\") pod \"network-check-target-gzq27\" (UID: \"6a2f0fa5-cbce-4695-aa38-e8bbeb720280\") " pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:32:33.904376 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.904354    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f4gw\" (UniqueName: \"kubernetes.io/projected/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-kube-api-access-4f4gw\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:32:33.905327 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.905309    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpswh\" (UniqueName: \"kubernetes.io/projected/0e0fdbea-232a-4a5e-abe4-3869da804c72-kube-api-access-gpswh\") pod \"alertmanager-main-0\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:33.905501 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.905484    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkc2j\" (UniqueName: \"kubernetes.io/projected/792b1392-5f44-47a8-9b86-05ee811046ea-kube-api-access-pkc2j\") pod \"prometheus-operator-9cd89b74c-8zqhk\" (UID: \"792b1392-5f44-47a8-9b86-05ee811046ea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8zqhk"
Nov 02 01:32:33.906380 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.906360    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zkz5\" (UniqueName: \"kubernetes.io/projected/f86263ab-30e1-40bf-a671-f82f44423125-kube-api-access-4zkz5\") pod \"openshift-state-metrics-7bd7489957-4snk7\" (UID: \"f86263ab-30e1-40bf-a671-f82f44423125\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4snk7"
Nov 02 01:32:33.906497 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.906480    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vmxb\" (UniqueName: \"kubernetes.io/projected/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-kube-api-access-5vmxb\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:32:33.945303 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945276    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-sysctl-d\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.945461 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945314    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-cnibin\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:32:33.945461 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945337    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-cnibin\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.945461 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945361    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-crio-socket\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:33.945461 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945381    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-var-lib-kubelet\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.945461 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945405    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-sys\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:33.945461 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945402    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-cnibin\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:32:33.945461 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945425    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-wtmp\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:33.945461 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945426    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-cnibin\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.945461 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945440    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59-crio-socket\") pod \"insights-runtime-extractor-r4njj\" (UID: \"2fbc2bb9-ce73-41fe-b36f-39c4e3ea4a59\") " pod="openshift-insights/insights-runtime-extractor-r4njj"
Nov 02 01:32:33.945461 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945456    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-var-lib-kubelet\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945462    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-sys\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945480    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-sysctl-d\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945498    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-node-exporter-wtmp\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945485    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-systemd\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945520    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-systemd\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945537    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-multus-conf-dir\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945564    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-multus-cni-dir\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945589    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-systemd-units\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945612    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-slash\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945620    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-multus-conf-dir\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945635    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-etc-openvswitch\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945644    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-slash\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945647    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-systemd-units\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945651    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-multus-cni-dir\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945663    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-cni-netd\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945677    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-etc-openvswitch\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945690    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b88b4dfb-1523-4701-9388-781cedab823b-dbus\") pod \"global-pull-secret-syncer-8vhjc\" (UID: \"b88b4dfb-1523-4701-9388-781cedab823b\") " pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945725    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-cni-netd\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945746    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-run-openvswitch\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945757    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b88b4dfb-1523-4701-9388-781cedab823b-dbus\") pod \"global-pull-secret-syncer-8vhjc\" (UID: \"b88b4dfb-1523-4701-9388-781cedab823b\") " pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945765    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-var-lib-kubelet\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945784    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-host\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945790    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-run-openvswitch\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945800    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-sys-fs\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945805    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-var-lib-kubelet\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945821    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-kubernetes\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945836    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b88b4dfb-1523-4701-9388-781cedab823b-kubelet-config\") pod \"global-pull-secret-syncer-8vhjc\" (UID: \"b88b4dfb-1523-4701-9388-781cedab823b\") " pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945846    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-sys-fs\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945858    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945838    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-host\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945876    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-kubernetes\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945884    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b88b4dfb-1523-4701-9388-781cedab823b-kubelet-config\") pod \"global-pull-secret-syncer-8vhjc\" (UID: \"b88b4dfb-1523-4701-9388-781cedab823b\") " pod="kube-system/global-pull-secret-syncer-8vhjc"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945878    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-device-dir\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945911    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-run-multus-certs\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945916    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945917    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-device-dir\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945931    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945950    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/adb8a888-520e-4ef5-819a-ee9331b0ccd9-host-slash\") pod \"iptables-alerter-f297t\" (UID: \"adb8a888-520e-4ef5-819a-ee9331b0ccd9\") " pod="openshift-network-operator/iptables-alerter-f297t"
Nov 02 01:32:33.945938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945952    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-run-multus-certs\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945974    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-run-netns\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945978    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/adb8a888-520e-4ef5-819a-ee9331b0ccd9-host-slash\") pod \"iptables-alerter-f297t\" (UID: \"adb8a888-520e-4ef5-819a-ee9331b0ccd9\") " pod="openshift-network-operator/iptables-alerter-f297t"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945985    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.945992    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-kubelet\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946017    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-kubelet\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946025    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-cni-bin\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946002    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-run-netns\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946052    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-registration-dir\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946078    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-etc-kubernetes\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946082    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-cni-bin\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946102    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-log-socket\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946115    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-registration-dir\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946124    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-sysconfig\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946142    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-etc-kubernetes\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946149    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-log-socket\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946175    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-sysconfig\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946265    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9f798cfa-1160-4da5-859c-47d81c07f969-hosts-file\") pod \"node-resolver-bd696\" (UID: \"9f798cfa-1160-4da5-859c-47d81c07f969\") " pod="openshift-dns/node-resolver-bd696"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946149    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9f798cfa-1160-4da5-859c-47d81c07f969-hosts-file\") pod \"node-resolver-bd696\" (UID: \"9f798cfa-1160-4da5-859c-47d81c07f969\") " pod="openshift-dns/node-resolver-bd696"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946683    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-sysctl-conf\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946712    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-system-cni-dir\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946734    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-run-systemd\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946759    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-var-lib-cni-bin\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946787    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-sys\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946790    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-run-systemd\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946795    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-system-cni-dir\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946810    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-run-k8s-cni-cncf-io\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946819    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-sysctl-conf\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946832    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-var-lib-cni-bin\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946838    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-socket-dir\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946845    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-sys\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946854    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-run-k8s-cni-cncf-io\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946860    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-etc-selinux\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946883    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-node-log\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946886    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-socket-dir\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946901    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8d19e74b-e519-4c0a-a3aa-956c6b3bcb28-etc-selinux\") pod \"aws-ebs-csi-driver-node-xtb8c\" (UID: \"8d19e74b-e519-4c0a-a3aa-956c6b3bcb28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtb8c"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946905    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-run\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946908    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-node-log\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946930    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946947    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-run\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946966    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-run-netns\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.946996    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-run-ovn\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.947012    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.947017    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-lib-modules\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.947038    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-run-ovn\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.947042    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-run-netns\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.947041    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-multus-socket-dir-parent\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.947067    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/046dd898-c779-4ce2-84ec-dc91afb8323f-host\") pod \"node-ca-7xs66\" (UID: \"046dd898-c779-4ce2-84ec-dc91afb8323f\") " pod="openshift-image-registry/node-ca-7xs66"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.947090    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-modprobe-d\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.947107    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-lib-modules\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.947120    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-var-lib-openvswitch\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.947126    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-multus-socket-dir-parent\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.947143    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-run-ovn-kubernetes\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.947147    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/046dd898-c779-4ce2-84ec-dc91afb8323f-host\") pod \"node-ca-7xs66\" (UID: \"046dd898-c779-4ce2-84ec-dc91afb8323f\") " pod="openshift-image-registry/node-ca-7xs66"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.947161    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-var-lib-openvswitch\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.947167    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/902d51a8-2407-4f4b-a5d9-134a40ad956c-etc-modprobe-d\") pod \"tuned-vh2dn\" (UID: \"902d51a8-2407-4f4b-a5d9-134a40ad956c\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2dn"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.947170    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-system-cni-dir\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.947191    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a39f3ef9-2d60-4da6-b69a-9bdfac039833-host-run-ovn-kubernetes\") pod \"ovnkube-node-gfkkb\" (UID: \"a39f3ef9-2d60-4da6-b69a-9bdfac039833\") " pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.947198    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-var-lib-cni-multus\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.947239    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6e02ef8d-7b7b-4de9-a2db-ae32d200a497-system-cni-dir\") pod \"multus-additional-cni-plugins-jtl6l\" (UID: \"6e02ef8d-7b7b-4de9-a2db-ae32d200a497\") " pod="openshift-multus/multus-additional-cni-plugins-jtl6l"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.947245    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-hostroot\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.947270    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-root\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.947248    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-host-var-lib-cni-multus\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.947319    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/32a38f9a-7d31-4a28-8a1b-8653ac3b7a20-hostroot\") pod \"multus-7tt8p\" (UID: \"32a38f9a-7d31-4a28-8a1b-8653ac3b7a20\") " pod="openshift-multus/multus-7tt8p"
Nov 02 01:32:33.947618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:33.947363    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9bbd7104-ba74-4c31-8cfb-3f307da4b3f4-root\") pod \"node-exporter-rrtpw\" (UID: \"9bbd7104-ba74-4c31-8cfb-3f307da4b3f4\") " pod="openshift-monitoring/node-exporter-rrtpw"
Nov 02 01:32:34.045818 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:34.045794    5875 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:32:34.919474 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:34.919442    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dbzkb"
Nov 02 01:32:35.987661 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:35.987629    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:36.105555 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:36.105519    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:36.111809 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:36.111782    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-b84687bcd-x24hf"
Nov 02 01:32:38.399245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:38.398819    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:38.802920 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:38.802882    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31ccc613-71f9-40b5-a1f9-9254f9e7598b","Type":"ContainerStarted","Data":"a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40"}
Nov 02 01:32:38.803103 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:38.803015    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31ccc613-71f9-40b5-a1f9-9254f9e7598b","Type":"ContainerStarted","Data":"f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838"}
Nov 02 01:32:38.803103 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:38.803052    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31ccc613-71f9-40b5-a1f9-9254f9e7598b","Type":"ContainerStarted","Data":"f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8"}
Nov 02 01:32:38.803103 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:38.803075    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31ccc613-71f9-40b5-a1f9-9254f9e7598b","Type":"ContainerStarted","Data":"0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f"}
Nov 02 01:32:38.803243 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:38.803153    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31ccc613-71f9-40b5-a1f9-9254f9e7598b","Type":"ContainerStarted","Data":"b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186"}
Nov 02 01:32:38.803243 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:38.803177    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31ccc613-71f9-40b5-a1f9-9254f9e7598b","Type":"ContainerStarted","Data":"9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490"}
Nov 02 01:32:40.294614 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:40.294518    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:40.295092 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:40.294759    5875 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 01:32:41.318333 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:41.318294    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:41.323547 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:32:41.323514    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:32:41.368645 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:41.368621    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:32:42.522406 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:42.522371    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:42.546418 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:42.546390    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:42.585521 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:42.585492    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kpwsv"
Nov 02 01:32:42.733173 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:42.733144    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gfkkb"
Nov 02 01:32:42.840558 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:42.840482    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:32:42.963212 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:42.963181    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rc9qm"
Nov 02 01:32:42.968451 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:42.968426    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rc9qm"
Nov 02 01:32:47.004440 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:47.004399    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:50.820488 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:50.820456    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ft5bg"
Nov 02 01:32:51.861372 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:51.861339    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-lltpm" event={"ID":"5dbcf879-26f8-4f7b-a071-eafa688b1c05","Type":"ContainerStarted","Data":"1fc72822155e272b7b0453f7ca9d638b9a225bcdedc4eea52c51abbd97c85a67"}
Nov 02 01:32:51.861780 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:51.861633    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-lltpm"
Nov 02 01:32:51.862885 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:51.862859    5875 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-lltpm container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.15:8080/\": dial tcp 10.134.0.15:8080: connect: connection refused" start-of-body=
Nov 02 01:32:51.863013 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:51.862911    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-lltpm" podUID="5dbcf879-26f8-4f7b-a071-eafa688b1c05" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.15:8080/\": dial tcp 10.134.0.15:8080: connect: connection refused"
Nov 02 01:32:52.877201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:52.877172    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-lltpm"
Nov 02 01:32:55.993506 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:55.993468    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:55.997925 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:55.997897    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-79ffdc6779-dp2ff"
Nov 02 01:32:56.162104 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:56.162068    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 01:32:56.162631 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:56.162594    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="alertmanager" containerID="cri-o://e2e645da2c8ae758a8ea9547107b8f095f5c550c1e2f328b17054d0b976a5389" gracePeriod=120
Nov 02 01:32:56.162912 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:56.162598    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="kube-rbac-proxy" containerID="cri-o://ad46a2edabcddbd3e9f15ce4d9894d049a0d5aaf9c2fc2a0b08db8e3c062726d" gracePeriod=120
Nov 02 01:32:56.163056 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:56.162615    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="kube-rbac-proxy-web" containerID="cri-o://586659c0f70929d188e32418856330c66b4904ae6e1749d108262ec1268263ee" gracePeriod=120
Nov 02 01:32:56.163056 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:56.162663    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="config-reloader" containerID="cri-o://857a6eb46b16acf31d19c387431a0d0aa1f82a690bd78fa7f93b6975f706bf4b" gracePeriod=120
Nov 02 01:32:56.163164 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:56.162670    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="kube-rbac-proxy-metric" containerID="cri-o://5850bdfb4ead613094576a77f4f9ef7385687a0e13c29cf7693a6dc0ef2f8398" gracePeriod=120
Nov 02 01:32:56.163242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:56.162730    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="prom-label-proxy" containerID="cri-o://bb400c45770004d2dcf8ff380f2d6208c9ee8b829dd62fb3e53f63b4ea3d38c2" gracePeriod=120
Nov 02 01:32:56.887376 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:56.887347    5875 generic.go:356] "Generic (PLEG): container finished" podID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerID="bb400c45770004d2dcf8ff380f2d6208c9ee8b829dd62fb3e53f63b4ea3d38c2" exitCode=0
Nov 02 01:32:56.887376 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:56.887372    5875 generic.go:356] "Generic (PLEG): container finished" podID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerID="5850bdfb4ead613094576a77f4f9ef7385687a0e13c29cf7693a6dc0ef2f8398" exitCode=0
Nov 02 01:32:56.887376 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:56.887379    5875 generic.go:356] "Generic (PLEG): container finished" podID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerID="ad46a2edabcddbd3e9f15ce4d9894d049a0d5aaf9c2fc2a0b08db8e3c062726d" exitCode=0
Nov 02 01:32:56.887376 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:56.887384    5875 generic.go:356] "Generic (PLEG): container finished" podID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerID="857a6eb46b16acf31d19c387431a0d0aa1f82a690bd78fa7f93b6975f706bf4b" exitCode=0
Nov 02 01:32:56.887376 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:56.887389    5875 generic.go:356] "Generic (PLEG): container finished" podID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerID="e2e645da2c8ae758a8ea9547107b8f095f5c550c1e2f328b17054d0b976a5389" exitCode=0
Nov 02 01:32:56.887860 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:56.887425    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e0fdbea-232a-4a5e-abe4-3869da804c72","Type":"ContainerDied","Data":"bb400c45770004d2dcf8ff380f2d6208c9ee8b829dd62fb3e53f63b4ea3d38c2"}
Nov 02 01:32:56.887860 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:56.887503    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e0fdbea-232a-4a5e-abe4-3869da804c72","Type":"ContainerDied","Data":"5850bdfb4ead613094576a77f4f9ef7385687a0e13c29cf7693a6dc0ef2f8398"}
Nov 02 01:32:56.887860 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:56.887533    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e0fdbea-232a-4a5e-abe4-3869da804c72","Type":"ContainerDied","Data":"ad46a2edabcddbd3e9f15ce4d9894d049a0d5aaf9c2fc2a0b08db8e3c062726d"}
Nov 02 01:32:56.887860 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:56.887566    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e0fdbea-232a-4a5e-abe4-3869da804c72","Type":"ContainerDied","Data":"857a6eb46b16acf31d19c387431a0d0aa1f82a690bd78fa7f93b6975f706bf4b"}
Nov 02 01:32:56.887860 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:56.887595    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e0fdbea-232a-4a5e-abe4-3869da804c72","Type":"ContainerDied","Data":"e2e645da2c8ae758a8ea9547107b8f095f5c550c1e2f328b17054d0b976a5389"}
Nov 02 01:32:57.898091 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:57.898056    5875 generic.go:356] "Generic (PLEG): container finished" podID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerID="586659c0f70929d188e32418856330c66b4904ae6e1749d108262ec1268263ee" exitCode=0
Nov 02 01:32:57.898576 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:57.898136    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e0fdbea-232a-4a5e-abe4-3869da804c72","Type":"ContainerDied","Data":"586659c0f70929d188e32418856330c66b4904ae6e1749d108262ec1268263ee"}
Nov 02 01:32:57.972123 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:57.972087    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:58.070488 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.070397    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e0fdbea-232a-4a5e-abe4-3869da804c72-tls-assets\" (UniqueName: \"kubernetes.io/projected/0e0fdbea-232a-4a5e-abe4-3869da804c72-tls-assets\") pod \"0e0fdbea-232a-4a5e-abe4-3869da804c72\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") "
Nov 02 01:32:58.070488 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.070443    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-cluster-tls-config\") pod \"0e0fdbea-232a-4a5e-abe4-3869da804c72\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") "
Nov 02 01:32:58.070488 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.070462    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy-metric\") pod \"0e0fdbea-232a-4a5e-abe4-3869da804c72\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") "
Nov 02 01:32:58.070784 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.070493    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e0fdbea-232a-4a5e-abe4-3869da804c72-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e0fdbea-232a-4a5e-abe4-3869da804c72-alertmanager-trusted-ca-bundle\") pod \"0e0fdbea-232a-4a5e-abe4-3869da804c72\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") "
Nov 02 01:32:58.070784 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.070537    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e0fdbea-232a-4a5e-abe4-3869da804c72-kube-api-access-gpswh\" (UniqueName: \"kubernetes.io/projected/0e0fdbea-232a-4a5e-abe4-3869da804c72-kube-api-access-gpswh\") pod \"0e0fdbea-232a-4a5e-abe4-3869da804c72\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") "
Nov 02 01:32:58.070784 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.070579    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0fdbea-232a-4a5e-abe4-3869da804c72-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0e0fdbea-232a-4a5e-abe4-3869da804c72-alertmanager-main-db\") pod \"0e0fdbea-232a-4a5e-abe4-3869da804c72\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") "
Nov 02 01:32:58.070784 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.070618    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e0fdbea-232a-4a5e-abe4-3869da804c72-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e0fdbea-232a-4a5e-abe4-3869da804c72-metrics-client-ca\") pod \"0e0fdbea-232a-4a5e-abe4-3869da804c72\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") "
Nov 02 01:32:58.070784 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.070650    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0fdbea-232a-4a5e-abe4-3869da804c72-config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e0fdbea-232a-4a5e-abe4-3869da804c72-config-out\") pod \"0e0fdbea-232a-4a5e-abe4-3869da804c72\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") "
Nov 02 01:32:58.070784 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.070693    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy-web\") pod \"0e0fdbea-232a-4a5e-abe4-3869da804c72\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") "
Nov 02 01:32:58.070784 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.070761    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-main-tls\") pod \"0e0fdbea-232a-4a5e-abe4-3869da804c72\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") "
Nov 02 01:32:58.071375 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.070804    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-config-volume\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-config-volume\") pod \"0e0fdbea-232a-4a5e-abe4-3869da804c72\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") "
Nov 02 01:32:58.071375 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.070833    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-web-config\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-web-config\") pod \"0e0fdbea-232a-4a5e-abe4-3869da804c72\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") "
Nov 02 01:32:58.071375 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.070875    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy\") pod \"0e0fdbea-232a-4a5e-abe4-3869da804c72\" (UID: \"0e0fdbea-232a-4a5e-abe4-3869da804c72\") "
Nov 02 01:32:58.071375 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.071103    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e0fdbea-232a-4a5e-abe4-3869da804c72-alertmanager-trusted-ca-bundle" pod "0e0fdbea-232a-4a5e-abe4-3869da804c72" (UID: "0e0fdbea-232a-4a5e-abe4-3869da804c72"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:32:58.071375 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.071165    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e0fdbea-232a-4a5e-abe4-3869da804c72-metrics-client-ca" pod "0e0fdbea-232a-4a5e-abe4-3869da804c72" (UID: "0e0fdbea-232a-4a5e-abe4-3869da804c72"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:32:58.071639 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.071411    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0fdbea-232a-4a5e-abe4-3869da804c72-alertmanager-main-db" pod "0e0fdbea-232a-4a5e-abe4-3869da804c72" (UID: "0e0fdbea-232a-4a5e-abe4-3869da804c72"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:32:58.072691 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.072334    5875 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e0fdbea-232a-4a5e-abe4-3869da804c72-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:58.072691 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.072364    5875 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0e0fdbea-232a-4a5e-abe4-3869da804c72-alertmanager-main-db\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:58.072691 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.072382    5875 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e0fdbea-232a-4a5e-abe4-3869da804c72-metrics-client-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:58.076125 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.075423    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-config-volume" pod "0e0fdbea-232a-4a5e-abe4-3869da804c72" (UID: "0e0fdbea-232a-4a5e-abe4-3869da804c72"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:32:58.076125 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.075708    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0fdbea-232a-4a5e-abe4-3869da804c72-config-out" pod "0e0fdbea-232a-4a5e-abe4-3869da804c72" (UID: "0e0fdbea-232a-4a5e-abe4-3869da804c72"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:32:58.076952 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.076911    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e0fdbea-232a-4a5e-abe4-3869da804c72-tls-assets" pod "0e0fdbea-232a-4a5e-abe4-3869da804c72" (UID: "0e0fdbea-232a-4a5e-abe4-3869da804c72"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:32:58.077296 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.077259    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy" pod "0e0fdbea-232a-4a5e-abe4-3869da804c72" (UID: "0e0fdbea-232a-4a5e-abe4-3869da804c72"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:32:58.077660 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.077621    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e0fdbea-232a-4a5e-abe4-3869da804c72-kube-api-access-gpswh" pod "0e0fdbea-232a-4a5e-abe4-3869da804c72" (UID: "0e0fdbea-232a-4a5e-abe4-3869da804c72"). InnerVolumeSpecName "kube-api-access-gpswh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:32:58.077762 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.077714    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy-web" pod "0e0fdbea-232a-4a5e-abe4-3869da804c72" (UID: "0e0fdbea-232a-4a5e-abe4-3869da804c72"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:32:58.079516 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.079482    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-main-tls" pod "0e0fdbea-232a-4a5e-abe4-3869da804c72" (UID: "0e0fdbea-232a-4a5e-abe4-3869da804c72"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:32:58.079735 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.079712    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy-metric" pod "0e0fdbea-232a-4a5e-abe4-3869da804c72" (UID: "0e0fdbea-232a-4a5e-abe4-3869da804c72"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:32:58.082512 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.082486    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-cluster-tls-config" pod "0e0fdbea-232a-4a5e-abe4-3869da804c72" (UID: "0e0fdbea-232a-4a5e-abe4-3869da804c72"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:32:58.090472 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.090447    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-web-config" pod "0e0fdbea-232a-4a5e-abe4-3869da804c72" (UID: "0e0fdbea-232a-4a5e-abe4-3869da804c72"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:32:58.173060 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.173019    5875 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:58.173060 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.173060    5875 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-main-tls\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:58.173310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.173074    5875 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:58.173310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.173087    5875 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-web-config\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:58.173310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.173099    5875 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:58.173310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.173113    5875 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e0fdbea-232a-4a5e-abe4-3869da804c72-tls-assets\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:58.173310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.173125    5875 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-cluster-tls-config\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:58.173310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.173137    5875 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0e0fdbea-232a-4a5e-abe4-3869da804c72-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:58.173310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.173151    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpswh\" (UniqueName: \"kubernetes.io/projected/0e0fdbea-232a-4a5e-abe4-3869da804c72-kube-api-access-gpswh\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:58.173310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.173163    5875 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e0fdbea-232a-4a5e-abe4-3869da804c72-config-out\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:58.906724 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.906684    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e0fdbea-232a-4a5e-abe4-3869da804c72","Type":"ContainerDied","Data":"b178ceb687ed740a6de9b4f920ff574b54f851f0634ea61f239e6902bce2af03"}
Nov 02 01:32:58.907203 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.906743    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:58.907203 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.906799    5875 scope.go:122] "RemoveContainer" containerID="bb400c45770004d2dcf8ff380f2d6208c9ee8b829dd62fb3e53f63b4ea3d38c2"
Nov 02 01:32:58.926985 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.926954    5875 scope.go:122] "RemoveContainer" containerID="5850bdfb4ead613094576a77f4f9ef7385687a0e13c29cf7693a6dc0ef2f8398"
Nov 02 01:32:58.935467 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.935446    5875 scope.go:122] "RemoveContainer" containerID="ad46a2edabcddbd3e9f15ce4d9894d049a0d5aaf9c2fc2a0b08db8e3c062726d"
Nov 02 01:32:58.943807 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.943787    5875 scope.go:122] "RemoveContainer" containerID="586659c0f70929d188e32418856330c66b4904ae6e1749d108262ec1268263ee"
Nov 02 01:32:58.951687 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.951668    5875 scope.go:122] "RemoveContainer" containerID="857a6eb46b16acf31d19c387431a0d0aa1f82a690bd78fa7f93b6975f706bf4b"
Nov 02 01:32:58.960062 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.960034    5875 scope.go:122] "RemoveContainer" containerID="e2e645da2c8ae758a8ea9547107b8f095f5c550c1e2f328b17054d0b976a5389"
Nov 02 01:32:58.968906 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:58.968883    5875 scope.go:122] "RemoveContainer" containerID="3ee78722ab1691cad1750ad559dc90bcb26c8b3c3120835a012d2d0e70cbeeba"
Nov 02 01:32:59.122959 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.122923    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 01:32:59.172735 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.172704    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 01:32:59.173451 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.173425    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 01:32:59.173815 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.173792    5875 cpu_manager.go:414] "containerMap: removing container" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="kube-rbac-proxy"
Nov 02 01:32:59.173815 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.173810    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="kube-rbac-proxy"
Nov 02 01:32:59.173968 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.173822    5875 cpu_manager.go:414] "containerMap: removing container" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="init-config-reloader"
Nov 02 01:32:59.173968 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.173828    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="init-config-reloader"
Nov 02 01:32:59.173968 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.173842    5875 cpu_manager.go:414] "containerMap: removing container" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="config-reloader"
Nov 02 01:32:59.173968 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.173849    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="config-reloader"
Nov 02 01:32:59.173968 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.173866    5875 cpu_manager.go:414] "containerMap: removing container" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="prom-label-proxy"
Nov 02 01:32:59.173968 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.173871    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="prom-label-proxy"
Nov 02 01:32:59.173968 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.173884    5875 cpu_manager.go:414] "containerMap: removing container" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="alertmanager"
Nov 02 01:32:59.173968 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.173890    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="alertmanager"
Nov 02 01:32:59.173968 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.173911    5875 cpu_manager.go:414] "containerMap: removing container" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="kube-rbac-proxy-metric"
Nov 02 01:32:59.173968 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.173916    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="kube-rbac-proxy-metric"
Nov 02 01:32:59.173968 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.173927    5875 cpu_manager.go:414] "containerMap: removing container" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="kube-rbac-proxy-web"
Nov 02 01:32:59.173968 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.173935    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="kube-rbac-proxy-web"
Nov 02 01:32:59.174452 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.174005    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="prom-label-proxy"
Nov 02 01:32:59.174452 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.174017    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="alertmanager"
Nov 02 01:32:59.174452 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.174024    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="kube-rbac-proxy-metric"
Nov 02 01:32:59.174452 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.174032    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="kube-rbac-proxy-web"
Nov 02 01:32:59.174452 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.174043    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="kube-rbac-proxy"
Nov 02 01:32:59.174452 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.174052    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="init-config-reloader"
Nov 02 01:32:59.174452 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.174064    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" containerName="config-reloader"
Nov 02 01:32:59.197000 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.196972    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.251016 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.250975    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 02 01:32:59.251183 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.251059    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 02 01:32:59.251183 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.251109    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 02 01:32:59.251183 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.251175    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 02 01:32:59.251350 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.251288    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 02 01:32:59.251395 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.251382    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 02 01:32:59.251444 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.251437    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 02 01:32:59.251551 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.251537    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-grbp4\""
Nov 02 01:32:59.251828 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.251809    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 02 01:32:59.283400 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.283369    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b1d7626a-fd60-4593-8a06-8740ed75893f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.283587 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.283410    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1d7626a-fd60-4593-8a06-8740ed75893f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.283587 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.283440    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1d7626a-fd60-4593-8a06-8740ed75893f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.283587 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.283503    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1d7626a-fd60-4593-8a06-8740ed75893f-web-config\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.283756 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.283636    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1d7626a-fd60-4593-8a06-8740ed75893f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.283756 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.283663    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1d7626a-fd60-4593-8a06-8740ed75893f-config-out\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.283836 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.283761    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b1d7626a-fd60-4593-8a06-8740ed75893f-config-volume\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.283836 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.283809    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b1d7626a-fd60-4593-8a06-8740ed75893f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.283924 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.283847    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnm8g\" (UniqueName: \"kubernetes.io/projected/b1d7626a-fd60-4593-8a06-8740ed75893f-kube-api-access-dnm8g\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.283924 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.283910    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b1d7626a-fd60-4593-8a06-8740ed75893f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.284017 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.283934    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b1d7626a-fd60-4593-8a06-8740ed75893f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.284017 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.283958    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1d7626a-fd60-4593-8a06-8740ed75893f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.284017 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.283987    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1d7626a-fd60-4593-8a06-8740ed75893f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.299580 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.299554    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 02 01:32:59.384587 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.384549    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnm8g\" (UniqueName: \"kubernetes.io/projected/b1d7626a-fd60-4593-8a06-8740ed75893f-kube-api-access-dnm8g\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.384789 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.384598    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b1d7626a-fd60-4593-8a06-8740ed75893f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.384789 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.384657    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b1d7626a-fd60-4593-8a06-8740ed75893f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.384789 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.384725    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1d7626a-fd60-4593-8a06-8740ed75893f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.384957 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.384794    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1d7626a-fd60-4593-8a06-8740ed75893f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.384957 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.384861    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b1d7626a-fd60-4593-8a06-8740ed75893f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.384957 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.384889    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1d7626a-fd60-4593-8a06-8740ed75893f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.384957 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.384920    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1d7626a-fd60-4593-8a06-8740ed75893f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.384957 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.384950    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1d7626a-fd60-4593-8a06-8740ed75893f-web-config\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.385201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.384978    5875 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 01:32:59.385201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.385002    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1d7626a-fd60-4593-8a06-8740ed75893f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.385201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.385033    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1d7626a-fd60-4593-8a06-8740ed75893f-config-out\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.385201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.385062    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b1d7626a-fd60-4593-8a06-8740ed75893f-config-volume\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.385201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.385089    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b1d7626a-fd60-4593-8a06-8740ed75893f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.385547 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.385521    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b1d7626a-fd60-4593-8a06-8740ed75893f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.386136 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.386114    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1d7626a-fd60-4593-8a06-8740ed75893f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.388013 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.387977    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1d7626a-fd60-4593-8a06-8740ed75893f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.388888 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.388804    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b1d7626a-fd60-4593-8a06-8740ed75893f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.389639 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.389595    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1d7626a-fd60-4593-8a06-8740ed75893f-config-out\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.391138 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.391107    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b1d7626a-fd60-4593-8a06-8740ed75893f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.391375 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.391352    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b1d7626a-fd60-4593-8a06-8740ed75893f-config-volume\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.392131 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.392108    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1d7626a-fd60-4593-8a06-8740ed75893f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.392575 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.392532    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1d7626a-fd60-4593-8a06-8740ed75893f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.394675 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.394654    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1d7626a-fd60-4593-8a06-8740ed75893f-web-config\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.395408 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.395379    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1d7626a-fd60-4593-8a06-8740ed75893f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.396522 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.396503    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b1d7626a-fd60-4593-8a06-8740ed75893f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.411408 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.411343    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 01:32:59.453180 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.453146    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnm8g\" (UniqueName: \"kubernetes.io/projected/b1d7626a-fd60-4593-8a06-8740ed75893f-kube-api-access-dnm8g\") pod \"alertmanager-main-0\" (UID: \"b1d7626a-fd60-4593-8a06-8740ed75893f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.679826 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.679780    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 01:32:59.922583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.922556    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5b76c759df-p6cl8"]
Nov 02 01:32:59.954698 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.954630    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:32:59.990480 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:32:59.990449    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b76c759df-p6cl8"]
Nov 02 01:33:00.007143 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.007108    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 02 01:33:00.007355 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.007163    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 02 01:33:00.007355 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.007166    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 02 01:33:00.007355 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.007207    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 02 01:33:00.007508 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.007395    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-br8w4\""
Nov 02 01:33:00.008531 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.008509    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 02 01:33:00.057127 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.057089    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 02 01:33:00.091186 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.091149    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9df8588e-97f8-4931-ad82-a5b4b2b02407-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b76c759df-p6cl8\" (UID: \"9df8588e-97f8-4931-ad82-a5b4b2b02407\") " pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:33:00.091385 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.091198    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9df8588e-97f8-4931-ad82-a5b4b2b02407-secret-telemeter-client\") pod \"telemeter-client-5b76c759df-p6cl8\" (UID: \"9df8588e-97f8-4931-ad82-a5b4b2b02407\") " pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:33:00.091385 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.091270    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn2fc\" (UniqueName: \"kubernetes.io/projected/9df8588e-97f8-4931-ad82-a5b4b2b02407-kube-api-access-hn2fc\") pod \"telemeter-client-5b76c759df-p6cl8\" (UID: \"9df8588e-97f8-4931-ad82-a5b4b2b02407\") " pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:33:00.091385 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.091316    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9df8588e-97f8-4931-ad82-a5b4b2b02407-metrics-client-ca\") pod \"telemeter-client-5b76c759df-p6cl8\" (UID: \"9df8588e-97f8-4931-ad82-a5b4b2b02407\") " pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:33:00.091385 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.091350    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9df8588e-97f8-4931-ad82-a5b4b2b02407-serving-certs-ca-bundle\") pod \"telemeter-client-5b76c759df-p6cl8\" (UID: \"9df8588e-97f8-4931-ad82-a5b4b2b02407\") " pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:33:00.091598 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.091402    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9df8588e-97f8-4931-ad82-a5b4b2b02407-federate-client-tls\") pod \"telemeter-client-5b76c759df-p6cl8\" (UID: \"9df8588e-97f8-4931-ad82-a5b4b2b02407\") " pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:33:00.091598 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.091538    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9df8588e-97f8-4931-ad82-a5b4b2b02407-telemeter-client-tls\") pod \"telemeter-client-5b76c759df-p6cl8\" (UID: \"9df8588e-97f8-4931-ad82-a5b4b2b02407\") " pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:33:00.091598 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.091574    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9df8588e-97f8-4931-ad82-a5b4b2b02407-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b76c759df-p6cl8\" (UID: \"9df8588e-97f8-4931-ad82-a5b4b2b02407\") " pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:33:00.191804 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.190874    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 01:33:00.192935 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.192905    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9df8588e-97f8-4931-ad82-a5b4b2b02407-federate-client-tls\") pod \"telemeter-client-5b76c759df-p6cl8\" (UID: \"9df8588e-97f8-4931-ad82-a5b4b2b02407\") " pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:33:00.193025 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.192964    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9df8588e-97f8-4931-ad82-a5b4b2b02407-telemeter-client-tls\") pod \"telemeter-client-5b76c759df-p6cl8\" (UID: \"9df8588e-97f8-4931-ad82-a5b4b2b02407\") " pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:33:00.193025 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.192994    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9df8588e-97f8-4931-ad82-a5b4b2b02407-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b76c759df-p6cl8\" (UID: \"9df8588e-97f8-4931-ad82-a5b4b2b02407\") " pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:33:00.193136 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.193026    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9df8588e-97f8-4931-ad82-a5b4b2b02407-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b76c759df-p6cl8\" (UID: \"9df8588e-97f8-4931-ad82-a5b4b2b02407\") " pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:33:00.193136 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.193046    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9df8588e-97f8-4931-ad82-a5b4b2b02407-secret-telemeter-client\") pod \"telemeter-client-5b76c759df-p6cl8\" (UID: \"9df8588e-97f8-4931-ad82-a5b4b2b02407\") " pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:33:00.193286 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.193234    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn2fc\" (UniqueName: \"kubernetes.io/projected/9df8588e-97f8-4931-ad82-a5b4b2b02407-kube-api-access-hn2fc\") pod \"telemeter-client-5b76c759df-p6cl8\" (UID: \"9df8588e-97f8-4931-ad82-a5b4b2b02407\") " pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:33:00.193342 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.193314    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9df8588e-97f8-4931-ad82-a5b4b2b02407-metrics-client-ca\") pod \"telemeter-client-5b76c759df-p6cl8\" (UID: \"9df8588e-97f8-4931-ad82-a5b4b2b02407\") " pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:33:00.193397 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.193348    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9df8588e-97f8-4931-ad82-a5b4b2b02407-serving-certs-ca-bundle\") pod \"telemeter-client-5b76c759df-p6cl8\" (UID: \"9df8588e-97f8-4931-ad82-a5b4b2b02407\") " pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:33:00.194172 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.194145    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9df8588e-97f8-4931-ad82-a5b4b2b02407-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b76c759df-p6cl8\" (UID: \"9df8588e-97f8-4931-ad82-a5b4b2b02407\") " pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:33:00.194325 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.194242    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9df8588e-97f8-4931-ad82-a5b4b2b02407-serving-certs-ca-bundle\") pod \"telemeter-client-5b76c759df-p6cl8\" (UID: \"9df8588e-97f8-4931-ad82-a5b4b2b02407\") " pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:33:00.194802 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.194778    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9df8588e-97f8-4931-ad82-a5b4b2b02407-metrics-client-ca\") pod \"telemeter-client-5b76c759df-p6cl8\" (UID: \"9df8588e-97f8-4931-ad82-a5b4b2b02407\") " pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:33:00.197324 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.197143    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9df8588e-97f8-4931-ad82-a5b4b2b02407-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b76c759df-p6cl8\" (UID: \"9df8588e-97f8-4931-ad82-a5b4b2b02407\") " pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:33:00.197706 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.197431    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9df8588e-97f8-4931-ad82-a5b4b2b02407-secret-telemeter-client\") pod \"telemeter-client-5b76c759df-p6cl8\" (UID: \"9df8588e-97f8-4931-ad82-a5b4b2b02407\") " pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:33:00.197706 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.197668    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9df8588e-97f8-4931-ad82-a5b4b2b02407-federate-client-tls\") pod \"telemeter-client-5b76c759df-p6cl8\" (UID: \"9df8588e-97f8-4931-ad82-a5b4b2b02407\") " pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:33:00.198087 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.198064    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9df8588e-97f8-4931-ad82-a5b4b2b02407-telemeter-client-tls\") pod \"telemeter-client-5b76c759df-p6cl8\" (UID: \"9df8588e-97f8-4931-ad82-a5b4b2b02407\") " pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:33:00.254917 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.254886    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn2fc\" (UniqueName: \"kubernetes.io/projected/9df8588e-97f8-4931-ad82-a5b4b2b02407-kube-api-access-hn2fc\") pod \"telemeter-client-5b76c759df-p6cl8\" (UID: \"9df8588e-97f8-4931-ad82-a5b4b2b02407\") " pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:33:00.334694 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.334667    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8"
Nov 02 01:33:00.693171 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.693098    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e0fdbea-232a-4a5e-abe4-3869da804c72" path="/var/lib/kubelet/pods/0e0fdbea-232a-4a5e-abe4-3869da804c72/volumes"
Nov 02 01:33:00.787375 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.787344    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b76c759df-p6cl8"]
Nov 02 01:33:00.790300 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:33:00.790264    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df8588e_97f8_4931_ad82_a5b4b2b02407.slice/crio-992fb6f1eda47f4a5d512b97678bd24480476b4047cad4bea512c2fac83dab82 WatchSource:0}: Error finding container 992fb6f1eda47f4a5d512b97678bd24480476b4047cad4bea512c2fac83dab82: Status 404 returned error can't find the container with id 992fb6f1eda47f4a5d512b97678bd24480476b4047cad4bea512c2fac83dab82
Nov 02 01:33:00.921055 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.921022    5875 generic.go:356] "Generic (PLEG): container finished" podID="b1d7626a-fd60-4593-8a06-8740ed75893f" containerID="1c011ae49ff1ae02deaae7131892b32f2d866f6b09757352f9a9f02d132889c3" exitCode=0
Nov 02 01:33:00.921273 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.921118    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b1d7626a-fd60-4593-8a06-8740ed75893f","Type":"ContainerDied","Data":"1c011ae49ff1ae02deaae7131892b32f2d866f6b09757352f9a9f02d132889c3"}
Nov 02 01:33:00.921273 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.921204    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b1d7626a-fd60-4593-8a06-8740ed75893f","Type":"ContainerStarted","Data":"eda61d90e5500ff77aa907a20839dd6a69201fb25791cfb34e8ed815a541f2ec"}
Nov 02 01:33:00.922386 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:00.922354    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8" event={"ID":"9df8588e-97f8-4931-ad82-a5b4b2b02407","Type":"ContainerStarted","Data":"992fb6f1eda47f4a5d512b97678bd24480476b4047cad4bea512c2fac83dab82"}
Nov 02 01:33:01.372254 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:01.371797    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gzq27"
Nov 02 01:33:01.907322 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:01.906544    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 01:33:01.907322 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:01.906918    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="prometheus" containerID="cri-o://9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490" gracePeriod=600
Nov 02 01:33:01.907322 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:01.906943    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="kube-rbac-proxy" containerID="cri-o://f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838" gracePeriod=600
Nov 02 01:33:01.907322 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:01.907079    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="kube-rbac-proxy-web" containerID="cri-o://f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8" gracePeriod=600
Nov 02 01:33:01.907322 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:01.907117    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40" gracePeriod=600
Nov 02 01:33:01.907322 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:01.907131    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="thanos-sidecar" containerID="cri-o://0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f" gracePeriod=600
Nov 02 01:33:01.907322 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:01.907196    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="config-reloader" containerID="cri-o://b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186" gracePeriod=600
Nov 02 01:33:01.936347 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:01.936314    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b1d7626a-fd60-4593-8a06-8740ed75893f","Type":"ContainerStarted","Data":"3ce44b6f78353af5cb0c60ff97d0bc3105555587f8677a71f881123011d30727"}
Nov 02 01:33:01.936515 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:01.936399    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b1d7626a-fd60-4593-8a06-8740ed75893f","Type":"ContainerStarted","Data":"59643284c5977a3aa918bd1084369bb77920d7338b0743a43d171d68aebbf36f"}
Nov 02 01:33:01.936515 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:01.936439    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b1d7626a-fd60-4593-8a06-8740ed75893f","Type":"ContainerStarted","Data":"477990fa27499c5fcae00942a3358301e8287735ff24f9f9874039217402f7c9"}
Nov 02 01:33:01.936515 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:01.936475    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b1d7626a-fd60-4593-8a06-8740ed75893f","Type":"ContainerStarted","Data":"e85918de7e602e5c8dcabc37a7418213ec3fba88b5041172e34f384bff55cb00"}
Nov 02 01:33:01.936515 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:01.936508    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b1d7626a-fd60-4593-8a06-8740ed75893f","Type":"ContainerStarted","Data":"fcd79dbf6024ae9ce45dffd32e0b9997a8d932663cae1f1fec5f23ec6cd91308"}
Nov 02 01:33:01.936720 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:01.936544    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b1d7626a-fd60-4593-8a06-8740ed75893f","Type":"ContainerStarted","Data":"aee26add3e64701f823fc51436f16d8db92aad10591f3ffe0a884f236d667374"}
Nov 02 01:33:02.182612 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.182509    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.156796455 podStartE2EDuration="3.182487782s" totalImagesPullingTime="0s" totalInitContainerRuntime="25.691327ms" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:33:02.182308127 +0000 UTC m=+29.623344625" watchObservedRunningTime="2026-11-02 01:33:02.182487782 +0000 UTC m=+29.623524307"
Nov 02 01:33:02.226798 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.226771    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:02.313258 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.313203    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31ccc613-71f9-40b5-a1f9-9254f9e7598b-kube-api-access-qc7c6\" (UniqueName: \"kubernetes.io/projected/31ccc613-71f9-40b5-a1f9-9254f9e7598b-kube-api-access-qc7c6\") pod \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") "
Nov 02 01:33:02.313258 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.313264    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-metrics-client-ca\") pod \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") "
Nov 02 01:33:02.313509 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.313297    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") "
Nov 02 01:33:02.313509 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.313336    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31ccc613-71f9-40b5-a1f9-9254f9e7598b-tls-assets\" (UniqueName: \"kubernetes.io/projected/31ccc613-71f9-40b5-a1f9-9254f9e7598b-tls-assets\") pod \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") "
Nov 02 01:33:02.313509 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.313373    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-2\") pod \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") "
Nov 02 01:33:02.313509 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.313402    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-serving-certs-ca-bundle\") pod \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") "
Nov 02 01:33:02.313509 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.313426    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-metrics-client-certs\") pod \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") "
Nov 02 01:33:02.313509 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.313467    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-grpc-tls\") pod \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") "
Nov 02 01:33:02.313509 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.313503    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-tls\") pod \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") "
Nov 02 01:33:02.313860 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.313532    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-kubelet-serving-ca-bundle\") pod \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") "
Nov 02 01:33:02.313860 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.313581    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") "
Nov 02 01:33:02.313860 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.313612    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-thanos-prometheus-http-client-file\") pod \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") "
Nov 02 01:33:02.313860 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.313646    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-0\") pod \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") "
Nov 02 01:33:02.313860 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.313684    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-kube-rbac-proxy\") pod \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") "
Nov 02 01:33:02.313860 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.313714    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-1\") pod \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") "
Nov 02 01:33:02.313860 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.313767    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-web-config\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-web-config\") pod \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") "
Nov 02 01:33:02.313860 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.313801    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-db\") pod \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") "
Nov 02 01:33:02.313860 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.313824    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31ccc613-71f9-40b5-a1f9-9254f9e7598b-config-out\" (UniqueName: \"kubernetes.io/empty-dir/31ccc613-71f9-40b5-a1f9-9254f9e7598b-config-out\") pod \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") "
Nov 02 01:33:02.313860 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.313863    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-trusted-ca-bundle\") pod \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") "
Nov 02 01:33:02.314384 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.313891    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-config\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-config\") pod \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\" (UID: \"31ccc613-71f9-40b5-a1f9-9254f9e7598b\") "
Nov 02 01:33:02.314384 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.313893    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-2" pod "31ccc613-71f9-40b5-a1f9-9254f9e7598b" (UID: "31ccc613-71f9-40b5-a1f9-9254f9e7598b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:33:02.314384 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.314160    5875 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:02.314384 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.314197    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-serving-certs-ca-bundle" pod "31ccc613-71f9-40b5-a1f9-9254f9e7598b" (UID: "31ccc613-71f9-40b5-a1f9-9254f9e7598b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:33:02.315213 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.314957    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-1" pod "31ccc613-71f9-40b5-a1f9-9254f9e7598b" (UID: "31ccc613-71f9-40b5-a1f9-9254f9e7598b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:33:02.316824 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.315585    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-kubelet-serving-ca-bundle" pod "31ccc613-71f9-40b5-a1f9-9254f9e7598b" (UID: "31ccc613-71f9-40b5-a1f9-9254f9e7598b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:33:02.316824 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.315665    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-metrics-client-ca" pod "31ccc613-71f9-40b5-a1f9-9254f9e7598b" (UID: "31ccc613-71f9-40b5-a1f9-9254f9e7598b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:33:02.317144 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.317122    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-trusted-ca-bundle" pod "31ccc613-71f9-40b5-a1f9-9254f9e7598b" (UID: "31ccc613-71f9-40b5-a1f9-9254f9e7598b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:33:02.317541 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.317519    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-0" pod "31ccc613-71f9-40b5-a1f9-9254f9e7598b" (UID: "31ccc613-71f9-40b5-a1f9-9254f9e7598b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:33:02.317823 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.317795    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-db" pod "31ccc613-71f9-40b5-a1f9-9254f9e7598b" (UID: "31ccc613-71f9-40b5-a1f9-9254f9e7598b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:33:02.318387 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.318164    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-kube-rbac-proxy" pod "31ccc613-71f9-40b5-a1f9-9254f9e7598b" (UID: "31ccc613-71f9-40b5-a1f9-9254f9e7598b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:33:02.320164 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.320138    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ccc613-71f9-40b5-a1f9-9254f9e7598b-tls-assets" pod "31ccc613-71f9-40b5-a1f9-9254f9e7598b" (UID: "31ccc613-71f9-40b5-a1f9-9254f9e7598b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:33:02.320347 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.320171    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-tls" pod "31ccc613-71f9-40b5-a1f9-9254f9e7598b" (UID: "31ccc613-71f9-40b5-a1f9-9254f9e7598b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:33:02.320821 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.320786    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-kube-rbac-proxy-web" pod "31ccc613-71f9-40b5-a1f9-9254f9e7598b" (UID: "31ccc613-71f9-40b5-a1f9-9254f9e7598b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:33:02.321267 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.321116    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ccc613-71f9-40b5-a1f9-9254f9e7598b-config-out" pod "31ccc613-71f9-40b5-a1f9-9254f9e7598b" (UID: "31ccc613-71f9-40b5-a1f9-9254f9e7598b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:33:02.321470 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.321443    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-thanos-prometheus-http-client-file" pod "31ccc613-71f9-40b5-a1f9-9254f9e7598b" (UID: "31ccc613-71f9-40b5-a1f9-9254f9e7598b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:33:02.321654 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.321584    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-thanos-sidecar-tls" pod "31ccc613-71f9-40b5-a1f9-9254f9e7598b" (UID: "31ccc613-71f9-40b5-a1f9-9254f9e7598b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:33:02.321739 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.321694    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-grpc-tls" pod "31ccc613-71f9-40b5-a1f9-9254f9e7598b" (UID: "31ccc613-71f9-40b5-a1f9-9254f9e7598b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:33:02.321797 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.321763    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-config" pod "31ccc613-71f9-40b5-a1f9-9254f9e7598b" (UID: "31ccc613-71f9-40b5-a1f9-9254f9e7598b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:33:02.322214 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.322188    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ccc613-71f9-40b5-a1f9-9254f9e7598b-kube-api-access-qc7c6" pod "31ccc613-71f9-40b5-a1f9-9254f9e7598b" (UID: "31ccc613-71f9-40b5-a1f9-9254f9e7598b"). InnerVolumeSpecName "kube-api-access-qc7c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:33:02.323090 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.323063    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-metrics-client-certs" pod "31ccc613-71f9-40b5-a1f9-9254f9e7598b" (UID: "31ccc613-71f9-40b5-a1f9-9254f9e7598b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:33:02.334431 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.334404    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-web-config" pod "31ccc613-71f9-40b5-a1f9-9254f9e7598b" (UID: "31ccc613-71f9-40b5-a1f9-9254f9e7598b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:33:02.414682 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.414650    5875 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-web-config\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:02.414682 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.414683    5875 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-db\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:02.415181 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.414701    5875 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31ccc613-71f9-40b5-a1f9-9254f9e7598b-config-out\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:02.415181 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.414716    5875 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:02.415181 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.414730    5875 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-config\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:02.415181 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.414743    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc7c6\" (UniqueName: \"kubernetes.io/projected/31ccc613-71f9-40b5-a1f9-9254f9e7598b-kube-api-access-qc7c6\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:02.415181 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.414761    5875 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-metrics-client-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:02.415181 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.414773    5875 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:02.415181 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.414782    5875 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31ccc613-71f9-40b5-a1f9-9254f9e7598b-tls-assets\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:02.415181 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.414791    5875 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:02.415181 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.414800    5875 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-metrics-client-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:02.415181 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.414808    5875 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-grpc-tls\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:02.415181 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.414819    5875 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:02.415181 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.414832    5875 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:02.415181 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.414841    5875 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:02.415181 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.414852    5875 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:02.415181 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.414860    5875 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:02.415181 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.414870    5875 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31ccc613-71f9-40b5-a1f9-9254f9e7598b-secret-kube-rbac-proxy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:02.415181 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.414879    5875 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/31ccc613-71f9-40b5-a1f9-9254f9e7598b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:02.945774 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.945738    5875 generic.go:356] "Generic (PLEG): container finished" podID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerID="a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40" exitCode=0
Nov 02 01:33:02.945774 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.945771    5875 generic.go:356] "Generic (PLEG): container finished" podID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerID="f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838" exitCode=0
Nov 02 01:33:02.946135 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.945783    5875 generic.go:356] "Generic (PLEG): container finished" podID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerID="f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8" exitCode=0
Nov 02 01:33:02.946135 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.945793    5875 generic.go:356] "Generic (PLEG): container finished" podID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerID="0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f" exitCode=0
Nov 02 01:33:02.946135 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.945801    5875 generic.go:356] "Generic (PLEG): container finished" podID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerID="b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186" exitCode=0
Nov 02 01:33:02.946135 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.945809    5875 generic.go:356] "Generic (PLEG): container finished" podID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerID="9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490" exitCode=0
Nov 02 01:33:02.946135 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.945838    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:02.946135 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.945844    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31ccc613-71f9-40b5-a1f9-9254f9e7598b","Type":"ContainerDied","Data":"a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40"}
Nov 02 01:33:02.946135 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.945917    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31ccc613-71f9-40b5-a1f9-9254f9e7598b","Type":"ContainerDied","Data":"f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838"}
Nov 02 01:33:02.946135 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.945964    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31ccc613-71f9-40b5-a1f9-9254f9e7598b","Type":"ContainerDied","Data":"f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8"}
Nov 02 01:33:02.946135 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.946005    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31ccc613-71f9-40b5-a1f9-9254f9e7598b","Type":"ContainerDied","Data":"0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f"}
Nov 02 01:33:02.946135 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.946023    5875 scope.go:122] "RemoveContainer" containerID="a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40"
Nov 02 01:33:02.946135 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.946033    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31ccc613-71f9-40b5-a1f9-9254f9e7598b","Type":"ContainerDied","Data":"b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186"}
Nov 02 01:33:02.946135 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.946066    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31ccc613-71f9-40b5-a1f9-9254f9e7598b","Type":"ContainerDied","Data":"9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490"}
Nov 02 01:33:02.946135 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.946105    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"31ccc613-71f9-40b5-a1f9-9254f9e7598b","Type":"ContainerDied","Data":"eebedc5242e32bfe4d2b763cd48f8cedf8ea1af380030a8cdd9f1cfc83993024"}
Nov 02 01:33:02.959361 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.959337    5875 scope.go:122] "RemoveContainer" containerID="f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838"
Nov 02 01:33:02.968304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.968279    5875 scope.go:122] "RemoveContainer" containerID="f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8"
Nov 02 01:33:02.980679 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.980586    5875 scope.go:122] "RemoveContainer" containerID="0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f"
Nov 02 01:33:02.989252 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.989190    5875 scope.go:122] "RemoveContainer" containerID="b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186"
Nov 02 01:33:02.999560 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:02.999536    5875 scope.go:122] "RemoveContainer" containerID="9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490"
Nov 02 01:33:03.008915 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.008893    5875 scope.go:122] "RemoveContainer" containerID="21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262"
Nov 02 01:33:03.016325 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.016300    5875 scope.go:122] "RemoveContainer" containerID="a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40"
Nov 02 01:33:03.016636 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:33:03.016607    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40\": container with ID starting with a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40 not found: ID does not exist" containerID="a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40"
Nov 02 01:33:03.016701 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.016647    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40"} err="failed to get container status \"a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40\": rpc error: code = NotFound desc = could not find container \"a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40\": container with ID starting with a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40 not found: ID does not exist"
Nov 02 01:33:03.016701 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.016680    5875 scope.go:122] "RemoveContainer" containerID="f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838"
Nov 02 01:33:03.016964 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:33:03.016942    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838\": container with ID starting with f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838 not found: ID does not exist" containerID="f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838"
Nov 02 01:33:03.017202 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.016971    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838"} err="failed to get container status \"f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838\": rpc error: code = NotFound desc = could not find container \"f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838\": container with ID starting with f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838 not found: ID does not exist"
Nov 02 01:33:03.017202 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.016985    5875 scope.go:122] "RemoveContainer" containerID="f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8"
Nov 02 01:33:03.017374 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:33:03.017355    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8\": container with ID starting with f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8 not found: ID does not exist" containerID="f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8"
Nov 02 01:33:03.017467 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.017375    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8"} err="failed to get container status \"f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8\": rpc error: code = NotFound desc = could not find container \"f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8\": container with ID starting with f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8 not found: ID does not exist"
Nov 02 01:33:03.017467 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.017391    5875 scope.go:122] "RemoveContainer" containerID="0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f"
Nov 02 01:33:03.017792 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:33:03.017762    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f\": container with ID starting with 0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f not found: ID does not exist" containerID="0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f"
Nov 02 01:33:03.017792 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.017785    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f"} err="failed to get container status \"0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f\": rpc error: code = NotFound desc = could not find container \"0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f\": container with ID starting with 0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f not found: ID does not exist"
Nov 02 01:33:03.017929 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.017802    5875 scope.go:122] "RemoveContainer" containerID="b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186"
Nov 02 01:33:03.018126 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:33:03.018106    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186\": container with ID starting with b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186 not found: ID does not exist" containerID="b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186"
Nov 02 01:33:03.018196 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.018130    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186"} err="failed to get container status \"b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186\": rpc error: code = NotFound desc = could not find container \"b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186\": container with ID starting with b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186 not found: ID does not exist"
Nov 02 01:33:03.018196 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.018152    5875 scope.go:122] "RemoveContainer" containerID="9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490"
Nov 02 01:33:03.018450 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:33:03.018434    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490\": container with ID starting with 9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490 not found: ID does not exist" containerID="9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490"
Nov 02 01:33:03.018500 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.018452    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490"} err="failed to get container status \"9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490\": rpc error: code = NotFound desc = could not find container \"9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490\": container with ID starting with 9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490 not found: ID does not exist"
Nov 02 01:33:03.018500 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.018464    5875 scope.go:122] "RemoveContainer" containerID="21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262"
Nov 02 01:33:03.018729 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:33:03.018709    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262\": container with ID starting with 21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262 not found: ID does not exist" containerID="21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262"
Nov 02 01:33:03.018787 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.018734    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262"} err="failed to get container status \"21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262\": rpc error: code = NotFound desc = could not find container \"21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262\": container with ID starting with 21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262 not found: ID does not exist"
Nov 02 01:33:03.018787 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.018748    5875 scope.go:122] "RemoveContainer" containerID="a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40"
Nov 02 01:33:03.019088 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.019068    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40"} err="failed to get container status \"a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40\": rpc error: code = NotFound desc = could not find container \"a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40\": container with ID starting with a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40 not found: ID does not exist"
Nov 02 01:33:03.019088 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.019087    5875 scope.go:122] "RemoveContainer" containerID="f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838"
Nov 02 01:33:03.019386 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.019369    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838"} err="failed to get container status \"f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838\": rpc error: code = NotFound desc = could not find container \"f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838\": container with ID starting with f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838 not found: ID does not exist"
Nov 02 01:33:03.019386 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.019385    5875 scope.go:122] "RemoveContainer" containerID="f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8"
Nov 02 01:33:03.019601 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.019582    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8"} err="failed to get container status \"f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8\": rpc error: code = NotFound desc = could not find container \"f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8\": container with ID starting with f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8 not found: ID does not exist"
Nov 02 01:33:03.019601 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.019598    5875 scope.go:122] "RemoveContainer" containerID="0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f"
Nov 02 01:33:03.019925 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.019891    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f"} err="failed to get container status \"0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f\": rpc error: code = NotFound desc = could not find container \"0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f\": container with ID starting with 0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f not found: ID does not exist"
Nov 02 01:33:03.019925 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.019913    5875 scope.go:122] "RemoveContainer" containerID="b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186"
Nov 02 01:33:03.020308 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.020285    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186"} err="failed to get container status \"b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186\": rpc error: code = NotFound desc = could not find container \"b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186\": container with ID starting with b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186 not found: ID does not exist"
Nov 02 01:33:03.020308 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.020307    5875 scope.go:122] "RemoveContainer" containerID="9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490"
Nov 02 01:33:03.020773 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.020751    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490"} err="failed to get container status \"9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490\": rpc error: code = NotFound desc = could not find container \"9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490\": container with ID starting with 9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490 not found: ID does not exist"
Nov 02 01:33:03.020846 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.020775    5875 scope.go:122] "RemoveContainer" containerID="21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262"
Nov 02 01:33:03.021016 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.020999    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262"} err="failed to get container status \"21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262\": rpc error: code = NotFound desc = could not find container \"21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262\": container with ID starting with 21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262 not found: ID does not exist"
Nov 02 01:33:03.021016 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.021016    5875 scope.go:122] "RemoveContainer" containerID="a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40"
Nov 02 01:33:03.021252 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.021212    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40"} err="failed to get container status \"a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40\": rpc error: code = NotFound desc = could not find container \"a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40\": container with ID starting with a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40 not found: ID does not exist"
Nov 02 01:33:03.021252 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.021251    5875 scope.go:122] "RemoveContainer" containerID="f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838"
Nov 02 01:33:03.021509 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.021488    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838"} err="failed to get container status \"f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838\": rpc error: code = NotFound desc = could not find container \"f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838\": container with ID starting with f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838 not found: ID does not exist"
Nov 02 01:33:03.021509 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.021508    5875 scope.go:122] "RemoveContainer" containerID="f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8"
Nov 02 01:33:03.021792 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.021776    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8"} err="failed to get container status \"f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8\": rpc error: code = NotFound desc = could not find container \"f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8\": container with ID starting with f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8 not found: ID does not exist"
Nov 02 01:33:03.021859 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.021793    5875 scope.go:122] "RemoveContainer" containerID="0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f"
Nov 02 01:33:03.022047 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.022023    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f"} err="failed to get container status \"0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f\": rpc error: code = NotFound desc = could not find container \"0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f\": container with ID starting with 0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f not found: ID does not exist"
Nov 02 01:33:03.022047 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.022046    5875 scope.go:122] "RemoveContainer" containerID="b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186"
Nov 02 01:33:03.022334 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.022309    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186"} err="failed to get container status \"b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186\": rpc error: code = NotFound desc = could not find container \"b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186\": container with ID starting with b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186 not found: ID does not exist"
Nov 02 01:33:03.022403 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.022336    5875 scope.go:122] "RemoveContainer" containerID="9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490"
Nov 02 01:33:03.022614 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.022595    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490"} err="failed to get container status \"9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490\": rpc error: code = NotFound desc = could not find container \"9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490\": container with ID starting with 9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490 not found: ID does not exist"
Nov 02 01:33:03.022674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.022615    5875 scope.go:122] "RemoveContainer" containerID="21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262"
Nov 02 01:33:03.022917 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.022893    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262"} err="failed to get container status \"21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262\": rpc error: code = NotFound desc = could not find container \"21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262\": container with ID starting with 21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262 not found: ID does not exist"
Nov 02 01:33:03.022990 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.022917    5875 scope.go:122] "RemoveContainer" containerID="a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40"
Nov 02 01:33:03.023199 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.023179    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40"} err="failed to get container status \"a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40\": rpc error: code = NotFound desc = could not find container \"a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40\": container with ID starting with a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40 not found: ID does not exist"
Nov 02 01:33:03.023287 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.023199    5875 scope.go:122] "RemoveContainer" containerID="f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838"
Nov 02 01:33:03.023451 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.023432    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838"} err="failed to get container status \"f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838\": rpc error: code = NotFound desc = could not find container \"f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838\": container with ID starting with f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838 not found: ID does not exist"
Nov 02 01:33:03.023451 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.023451    5875 scope.go:122] "RemoveContainer" containerID="f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8"
Nov 02 01:33:03.023720 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.023700    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8"} err="failed to get container status \"f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8\": rpc error: code = NotFound desc = could not find container \"f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8\": container with ID starting with f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8 not found: ID does not exist"
Nov 02 01:33:03.023720 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.023720    5875 scope.go:122] "RemoveContainer" containerID="0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f"
Nov 02 01:33:03.023937 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.023922    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f"} err="failed to get container status \"0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f\": rpc error: code = NotFound desc = could not find container \"0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f\": container with ID starting with 0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f not found: ID does not exist"
Nov 02 01:33:03.023985 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.023938    5875 scope.go:122] "RemoveContainer" containerID="b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186"
Nov 02 01:33:03.024158 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.024140    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186"} err="failed to get container status \"b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186\": rpc error: code = NotFound desc = could not find container \"b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186\": container with ID starting with b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186 not found: ID does not exist"
Nov 02 01:33:03.024248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.024158    5875 scope.go:122] "RemoveContainer" containerID="9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490"
Nov 02 01:33:03.024397 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.024381    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490"} err="failed to get container status \"9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490\": rpc error: code = NotFound desc = could not find container \"9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490\": container with ID starting with 9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490 not found: ID does not exist"
Nov 02 01:33:03.024460 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.024397    5875 scope.go:122] "RemoveContainer" containerID="21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262"
Nov 02 01:33:03.024596 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.024578    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262"} err="failed to get container status \"21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262\": rpc error: code = NotFound desc = could not find container \"21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262\": container with ID starting with 21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262 not found: ID does not exist"
Nov 02 01:33:03.024658 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.024596    5875 scope.go:122] "RemoveContainer" containerID="a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40"
Nov 02 01:33:03.024868 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.024849    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40"} err="failed to get container status \"a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40\": rpc error: code = NotFound desc = could not find container \"a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40\": container with ID starting with a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40 not found: ID does not exist"
Nov 02 01:33:03.024948 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.024869    5875 scope.go:122] "RemoveContainer" containerID="f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838"
Nov 02 01:33:03.025125 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.025107    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838"} err="failed to get container status \"f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838\": rpc error: code = NotFound desc = could not find container \"f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838\": container with ID starting with f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838 not found: ID does not exist"
Nov 02 01:33:03.025206 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.025125    5875 scope.go:122] "RemoveContainer" containerID="f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8"
Nov 02 01:33:03.025377 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.025358    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8"} err="failed to get container status \"f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8\": rpc error: code = NotFound desc = could not find container \"f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8\": container with ID starting with f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8 not found: ID does not exist"
Nov 02 01:33:03.025449 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.025377    5875 scope.go:122] "RemoveContainer" containerID="0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f"
Nov 02 01:33:03.025595 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.025578    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f"} err="failed to get container status \"0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f\": rpc error: code = NotFound desc = could not find container \"0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f\": container with ID starting with 0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f not found: ID does not exist"
Nov 02 01:33:03.025662 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.025595    5875 scope.go:122] "RemoveContainer" containerID="b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186"
Nov 02 01:33:03.025842 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.025819    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186"} err="failed to get container status \"b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186\": rpc error: code = NotFound desc = could not find container \"b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186\": container with ID starting with b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186 not found: ID does not exist"
Nov 02 01:33:03.025904 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.025843    5875 scope.go:122] "RemoveContainer" containerID="9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490"
Nov 02 01:33:03.026099 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.026076    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490"} err="failed to get container status \"9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490\": rpc error: code = NotFound desc = could not find container \"9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490\": container with ID starting with 9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490 not found: ID does not exist"
Nov 02 01:33:03.026099 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.026098    5875 scope.go:122] "RemoveContainer" containerID="21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262"
Nov 02 01:33:03.026351 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.026329    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262"} err="failed to get container status \"21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262\": rpc error: code = NotFound desc = could not find container \"21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262\": container with ID starting with 21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262 not found: ID does not exist"
Nov 02 01:33:03.026351 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.026352    5875 scope.go:122] "RemoveContainer" containerID="a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40"
Nov 02 01:33:03.026594 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.026574    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40"} err="failed to get container status \"a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40\": rpc error: code = NotFound desc = could not find container \"a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40\": container with ID starting with a354c241214400b9f973a50d12c037ebd954e6e3d0e57e244c3da0c6345a0f40 not found: ID does not exist"
Nov 02 01:33:03.026651 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.026594    5875 scope.go:122] "RemoveContainer" containerID="f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838"
Nov 02 01:33:03.026851 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.026821    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838"} err="failed to get container status \"f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838\": rpc error: code = NotFound desc = could not find container \"f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838\": container with ID starting with f865ccc4bc33da6a9f4021885cb1b2b781bf3b36c3ab4f367e6f7cb37b537838 not found: ID does not exist"
Nov 02 01:33:03.026851 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.026844    5875 scope.go:122] "RemoveContainer" containerID="f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8"
Nov 02 01:33:03.027085 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.027053    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8"} err="failed to get container status \"f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8\": rpc error: code = NotFound desc = could not find container \"f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8\": container with ID starting with f42ea81afdb341405a95383a0b6c076d73e862919ad9b82bbfbfaabbecc240c8 not found: ID does not exist"
Nov 02 01:33:03.027085 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.027076    5875 scope.go:122] "RemoveContainer" containerID="0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f"
Nov 02 01:33:03.027335 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.027314    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f"} err="failed to get container status \"0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f\": rpc error: code = NotFound desc = could not find container \"0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f\": container with ID starting with 0302e391777eb24a50c6e90c9444263b55c1d8c9a672485d80e6fcf9dd68142f not found: ID does not exist"
Nov 02 01:33:03.027335 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.027333    5875 scope.go:122] "RemoveContainer" containerID="b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186"
Nov 02 01:33:03.027594 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.027574    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186"} err="failed to get container status \"b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186\": rpc error: code = NotFound desc = could not find container \"b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186\": container with ID starting with b5af326442f938bde3b920686735739272c9a7e271bcb98685261a6caa66d186 not found: ID does not exist"
Nov 02 01:33:03.027594 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.027592    5875 scope.go:122] "RemoveContainer" containerID="9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490"
Nov 02 01:33:03.027848 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.027825    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490"} err="failed to get container status \"9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490\": rpc error: code = NotFound desc = could not find container \"9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490\": container with ID starting with 9b136f4e7ddc49a2f261a2a659e4811c3e713159aec8e112230565494c374490 not found: ID does not exist"
Nov 02 01:33:03.027924 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.027850    5875 scope.go:122] "RemoveContainer" containerID="21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262"
Nov 02 01:33:03.028114 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.028094    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262"} err="failed to get container status \"21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262\": rpc error: code = NotFound desc = could not find container \"21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262\": container with ID starting with 21d27252a844dffe7028c40cb7cf4437546d083af60469bb4d4be472de3b5262 not found: ID does not exist"
Nov 02 01:33:03.112594 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.112553    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 01:33:03.162264 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.162183    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 01:33:03.163005 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.162972    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 01:33:03.163577 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.163548    5875 cpu_manager.go:414] "containerMap: removing container" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="config-reloader"
Nov 02 01:33:03.163577 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.163569    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="config-reloader"
Nov 02 01:33:03.163705 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.163588    5875 cpu_manager.go:414] "containerMap: removing container" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="kube-rbac-proxy-web"
Nov 02 01:33:03.163705 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.163597    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="kube-rbac-proxy-web"
Nov 02 01:33:03.163705 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.163608    5875 cpu_manager.go:414] "containerMap: removing container" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="kube-rbac-proxy-thanos"
Nov 02 01:33:03.163705 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.163615    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="kube-rbac-proxy-thanos"
Nov 02 01:33:03.163705 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.163630    5875 cpu_manager.go:414] "containerMap: removing container" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="prometheus"
Nov 02 01:33:03.163705 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.163637    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="prometheus"
Nov 02 01:33:03.163705 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.163664    5875 cpu_manager.go:414] "containerMap: removing container" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="kube-rbac-proxy"
Nov 02 01:33:03.163705 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.163671    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="kube-rbac-proxy"
Nov 02 01:33:03.163705 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.163687    5875 cpu_manager.go:414] "containerMap: removing container" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="init-config-reloader"
Nov 02 01:33:03.163705 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.163694    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="init-config-reloader"
Nov 02 01:33:03.164069 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.163711    5875 cpu_manager.go:414] "containerMap: removing container" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="thanos-sidecar"
Nov 02 01:33:03.164069 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.163719    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="thanos-sidecar"
Nov 02 01:33:03.164069 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.163773    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="thanos-sidecar"
Nov 02 01:33:03.164069 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.163782    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="config-reloader"
Nov 02 01:33:03.164069 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.163788    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="kube-rbac-proxy-web"
Nov 02 01:33:03.164069 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.163796    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="kube-rbac-proxy-thanos"
Nov 02 01:33:03.164069 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.163803    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="prometheus"
Nov 02 01:33:03.164069 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.163814    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" containerName="kube-rbac-proxy"
Nov 02 01:33:03.191049 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.191016    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.243725 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.243646    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 02 01:33:03.243725 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.243666    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 02 01:33:03.243984 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.243963    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 02 01:33:03.244052 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.244014    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 02 01:33:03.244052 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.244014    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 02 01:33:03.244169 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.244065    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 02 01:33:03.244169 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.244137    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 02 01:33:03.244169 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.244141    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 02 01:33:03.244388 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.244258    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 02 01:33:03.244388 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.244288    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 02 01:33:03.244388 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.244303    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 02 01:33:03.244388 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.244310    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 02 01:33:03.244566 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.244437    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-39jc004slgm3b\""
Nov 02 01:33:03.244566 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.244442    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-22x2k\""
Nov 02 01:33:03.294565 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.294531    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 02 01:33:03.294565 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.294537    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 02 01:33:03.295674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.295653    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 01:33:03.322871 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.322843    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-web-config\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.323046 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.322882    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.323046 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.322903    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.323046 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.323018    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.323197 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.323054    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.323197 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.323088    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.323197 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.323148    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.323317 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.323216    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.323317 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.323281    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.323389 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.323320    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.323389 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.323381    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-config\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.323446 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.323416    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.323446 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.323433    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5htm\" (UniqueName: \"kubernetes.io/projected/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-kube-api-access-x5htm\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.323521 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.323451    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.323521 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.323472    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.323613 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.323553    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.323613 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.323583    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.323613 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.323599    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-config-out\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.323704 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.323615    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.323704 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.323640    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.424584 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.424545    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.424584 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.424581    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.425087 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.424598    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.425087 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.424711    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.425087 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.424762    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.425087 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.424791    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.425087 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.424814    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.425087 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.424850    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.425087 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.424901    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.425087 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.424930    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-config\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.425087 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.424953    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.425087 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.424979    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5htm\" (UniqueName: \"kubernetes.io/projected/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-kube-api-access-x5htm\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.425087 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.425007    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.425087 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.425029    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.425087 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.425095    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.425789 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.425131    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.425789 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.425156    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-config-out\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.425789 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.425180    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.425789 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.425203    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.425789 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.425269    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-web-config\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.425789 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.425389    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.425789 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.425493    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.425789 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.425760    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.430295 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.426140    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.430295 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.426470    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.430295 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.427268    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.430295 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.427619    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.430295 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.428254    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.430295 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.429013    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.430295 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.429109    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.430295 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.430042    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-web-config\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.430900 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.430864    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.431474 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.431440    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.431673 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.431629    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.431673 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.431669    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.431937 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.431922    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-config-out\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.432414 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.432396    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-config\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.432861 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.432838    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.432980 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.432965    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.491703 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.491664    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5htm\" (UniqueName: \"kubernetes.io/projected/6625c4e9-083d-4c39-9bb8-e3398cc9f68b-kube-api-access-x5htm\") pod \"prometheus-k8s-0\" (UID: \"6625c4e9-083d-4c39-9bb8-e3398cc9f68b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.634852 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.634769    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:03.951826 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.951798    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8" event={"ID":"9df8588e-97f8-4931-ad82-a5b4b2b02407","Type":"ContainerStarted","Data":"bff1614f564d468673eb2e60b1a84b75bb38b45eba6b1b708b7c92fbf485bb8e"}
Nov 02 01:33:03.951958 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:03.951853    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8" event={"ID":"9df8588e-97f8-4931-ad82-a5b4b2b02407","Type":"ContainerStarted","Data":"05614dd3783fd4ed84cc3e125dd3e0de44cd36cf268732b7db0b8e6b2e8020a9"}
Nov 02 01:33:04.204523 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:04.204446    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 01:33:04.207407 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:33:04.207375    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6625c4e9_083d_4c39_9bb8_e3398cc9f68b.slice/crio-f9c01f62eb62d28f97203d38616547e4c91f14fe560f284db3a06f12935fec13 WatchSource:0}: Error finding container f9c01f62eb62d28f97203d38616547e4c91f14fe560f284db3a06f12935fec13: Status 404 returned error can't find the container with id f9c01f62eb62d28f97203d38616547e4c91f14fe560f284db3a06f12935fec13
Nov 02 01:33:04.692459 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:04.692426    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31ccc613-71f9-40b5-a1f9-9254f9e7598b" path="/var/lib/kubelet/pods/31ccc613-71f9-40b5-a1f9-9254f9e7598b/volumes"
Nov 02 01:33:04.960920 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:04.960887    5875 generic.go:356] "Generic (PLEG): container finished" podID="6625c4e9-083d-4c39-9bb8-e3398cc9f68b" containerID="c0d67af3d595680dcce6b75854717de9844b1b66479fcd817ac3a21981aa6742" exitCode=0
Nov 02 01:33:04.961322 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:04.960974    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6625c4e9-083d-4c39-9bb8-e3398cc9f68b","Type":"ContainerDied","Data":"c0d67af3d595680dcce6b75854717de9844b1b66479fcd817ac3a21981aa6742"}
Nov 02 01:33:04.961322 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:04.961088    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6625c4e9-083d-4c39-9bb8-e3398cc9f68b","Type":"ContainerStarted","Data":"f9c01f62eb62d28f97203d38616547e4c91f14fe560f284db3a06f12935fec13"}
Nov 02 01:33:04.963160 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:04.963137    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8" event={"ID":"9df8588e-97f8-4931-ad82-a5b4b2b02407","Type":"ContainerStarted","Data":"8fd4b748aa211f3e103fd2d1a0c67d90acc06d0b66467b16c9ca54b3d9407caf"}
Nov 02 01:33:05.215038 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:05.214940    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5b76c759df-p6cl8" podStartSLOduration=3.275987265 podStartE2EDuration="6.214924676s" totalImagesPullingTime="2.938937411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:33:05.21353024 +0000 UTC m=+32.654566738" watchObservedRunningTime="2026-11-02 01:33:05.214924676 +0000 UTC m=+32.655961170"
Nov 02 01:33:05.970287 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:05.970250    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6625c4e9-083d-4c39-9bb8-e3398cc9f68b","Type":"ContainerStarted","Data":"e9a9ea4e87ead51207bafc01f9d026ba042366f5d88219eb47ed975a7d0b307f"}
Nov 02 01:33:05.970749 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:05.970328    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6625c4e9-083d-4c39-9bb8-e3398cc9f68b","Type":"ContainerStarted","Data":"98fd498c2095665a3f81e1e09de94095527bdbe841af0c583ceb8dac43d20fe2"}
Nov 02 01:33:05.970749 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:05.970361    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6625c4e9-083d-4c39-9bb8-e3398cc9f68b","Type":"ContainerStarted","Data":"a0f1d26a4b74c629e463a912d0eb564f8228cbfed0844a509e70248982a39f6e"}
Nov 02 01:33:05.970749 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:05.970387    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6625c4e9-083d-4c39-9bb8-e3398cc9f68b","Type":"ContainerStarted","Data":"a8eb9b9ea32a3cfa910bd50f0e5720880a687574cc79620ed933eeeae56ba578"}
Nov 02 01:33:05.970749 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:05.970417    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6625c4e9-083d-4c39-9bb8-e3398cc9f68b","Type":"ContainerStarted","Data":"4750ca8f9512b6c21d6be067aa8aeb871ad56b66247d42006bc5ddecfd0f9ee5"}
Nov 02 01:33:05.970749 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:05.970438    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6625c4e9-083d-4c39-9bb8-e3398cc9f68b","Type":"ContainerStarted","Data":"a1eb75f43475f8fffa67e8707bed33080d6305ab9bbe7707ee0e842234b2899f"}
Nov 02 01:33:06.103630 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:06.103579    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.085921752 podStartE2EDuration="3.103563849s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.642097ms" isStatefulPod=false podCreationTimestamp="2026-11-02 01:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:33:06.102475496 +0000 UTC m=+33.543511994" watchObservedRunningTime="2026-11-02 01:33:06.103563849 +0000 UTC m=+33.544600341"
Nov 02 01:33:08.635573 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:08.635532    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:11.317600 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:33:11.317557    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:33:41.317375 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:33:41.317335    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:33:52.485116 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:52.485070    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:52.501707 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:52.501686    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:33:53.162039 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:33:53.162010    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:34:11.316995 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:34:11.316910    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:34:41.316795 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:34:41.316749    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:35:11.316134 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:35:11.316097    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:35:41.317060 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:35:41.316981    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:36:11.316933 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:36:11.316895    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:36:41.316739 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:36:41.316703    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:37:11.316866 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:37:11.316781    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:37:34.964514 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:34.964478    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-b47p5"]
Nov 02 01:37:34.966671 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:34.966655    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b47p5"
Nov 02 01:37:35.057653 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:35.057620    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9zlt\" (UniqueName: \"kubernetes.io/projected/334ade60-75f1-42c5-848a-eec507a0ab1f-kube-api-access-l9zlt\") pod \"jobset-operator-674cd59dc5-b47p5\" (UID: \"334ade60-75f1-42c5-848a-eec507a0ab1f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b47p5"
Nov 02 01:37:35.057806 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:35.057741    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/334ade60-75f1-42c5-848a-eec507a0ab1f-tmp\") pod \"jobset-operator-674cd59dc5-b47p5\" (UID: \"334ade60-75f1-42c5-848a-eec507a0ab1f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b47p5"
Nov 02 01:37:35.068593 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:35.068565    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-8crnj\""
Nov 02 01:37:35.068755 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:35.068591    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 02 01:37:35.070624 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:35.070598    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-b47p5"]
Nov 02 01:37:35.121939 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:35.121907    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:37:35.158841 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:35.158801    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/334ade60-75f1-42c5-848a-eec507a0ab1f-tmp\") pod \"jobset-operator-674cd59dc5-b47p5\" (UID: \"334ade60-75f1-42c5-848a-eec507a0ab1f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b47p5"
Nov 02 01:37:35.158841 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:35.158845    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9zlt\" (UniqueName: \"kubernetes.io/projected/334ade60-75f1-42c5-848a-eec507a0ab1f-kube-api-access-l9zlt\") pod \"jobset-operator-674cd59dc5-b47p5\" (UID: \"334ade60-75f1-42c5-848a-eec507a0ab1f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b47p5"
Nov 02 01:37:35.159178 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:35.159156    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/334ade60-75f1-42c5-848a-eec507a0ab1f-tmp\") pod \"jobset-operator-674cd59dc5-b47p5\" (UID: \"334ade60-75f1-42c5-848a-eec507a0ab1f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b47p5"
Nov 02 01:37:35.231551 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:35.231477    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9zlt\" (UniqueName: \"kubernetes.io/projected/334ade60-75f1-42c5-848a-eec507a0ab1f-kube-api-access-l9zlt\") pod \"jobset-operator-674cd59dc5-b47p5\" (UID: \"334ade60-75f1-42c5-848a-eec507a0ab1f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b47p5"
Nov 02 01:37:35.289614 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:35.289583    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b47p5"
Nov 02 01:37:35.659251 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:35.659148    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-b47p5"]
Nov 02 01:37:35.662279 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:37:35.662251    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod334ade60_75f1_42c5_848a_eec507a0ab1f.slice/crio-a268ce69b0843cb69b25cc907c9accd28d60601c3a3afa9d437f31980d0d029a WatchSource:0}: Error finding container a268ce69b0843cb69b25cc907c9accd28d60601c3a3afa9d437f31980d0d029a: Status 404 returned error can't find the container with id a268ce69b0843cb69b25cc907c9accd28d60601c3a3afa9d437f31980d0d029a
Nov 02 01:37:35.663864 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:35.663842    5875 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:37:35.959515 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:35.959482    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b47p5" event={"ID":"334ade60-75f1-42c5-848a-eec507a0ab1f","Type":"ContainerStarted","Data":"a268ce69b0843cb69b25cc907c9accd28d60601c3a3afa9d437f31980d0d029a"}
Nov 02 01:37:39.978464 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:39.978428    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b47p5" event={"ID":"334ade60-75f1-42c5-848a-eec507a0ab1f","Type":"ContainerStarted","Data":"fb5499d34c24fa9e1a63a41624355bf4e85559ef7d696c133a927fa6ef768406"}
Nov 02 01:37:40.094398 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:40.094345    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b47p5" podStartSLOduration=2.317691826 podStartE2EDuration="6.094331003s" totalImagesPullingTime="3.776639177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:40.093913874 +0000 UTC m=+307.534950372" watchObservedRunningTime="2026-11-02 01:37:40.094331003 +0000 UTC m=+307.535367499"
Nov 02 01:37:41.322149 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:37:41.322113    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:37:43.446618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.446589    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb"]
Nov 02 01:37:43.448779 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.448764    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb"
Nov 02 01:37:43.507040 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.507011    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 01:37:43.507204 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.507065    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 01:37:43.507840 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.507825    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 01:37:43.542520 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.542495    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f7374a7-4abb-48a2-9704-748bf75c84aa-tmp\") pod \"opendatahub-operator-controller-manager-6755fff5c-lr4lb\" (UID: \"3f7374a7-4abb-48a2-9704-748bf75c84aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb"
Nov 02 01:37:43.542520 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.542524    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3f7374a7-4abb-48a2-9704-748bf75c84aa-manifests\") pod \"opendatahub-operator-controller-manager-6755fff5c-lr4lb\" (UID: \"3f7374a7-4abb-48a2-9704-748bf75c84aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb"
Nov 02 01:37:43.542706 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.542559    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tzwb\" (UniqueName: \"kubernetes.io/projected/3f7374a7-4abb-48a2-9704-748bf75c84aa-kube-api-access-9tzwb\") pod \"opendatahub-operator-controller-manager-6755fff5c-lr4lb\" (UID: \"3f7374a7-4abb-48a2-9704-748bf75c84aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb"
Nov 02 01:37:43.542706 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.542618    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f7374a7-4abb-48a2-9704-748bf75c84aa-webhook-cert\") pod \"opendatahub-operator-controller-manager-6755fff5c-lr4lb\" (UID: \"3f7374a7-4abb-48a2-9704-748bf75c84aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb"
Nov 02 01:37:43.542770 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.542715    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3f7374a7-4abb-48a2-9704-748bf75c84aa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6755fff5c-lr4lb\" (UID: \"3f7374a7-4abb-48a2-9704-748bf75c84aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb"
Nov 02 01:37:43.542815 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.542776    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3f7374a7-4abb-48a2-9704-748bf75c84aa-metrics-cert\") pod \"opendatahub-operator-controller-manager-6755fff5c-lr4lb\" (UID: \"3f7374a7-4abb-48a2-9704-748bf75c84aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb"
Nov 02 01:37:43.552935 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.552911    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 01:37:43.553100 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.552993    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 01:37:43.553100 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.553031    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-gnd7s\""
Nov 02 01:37:43.554321 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.554300    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb"]
Nov 02 01:37:43.643422 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.643398    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tzwb\" (UniqueName: \"kubernetes.io/projected/3f7374a7-4abb-48a2-9704-748bf75c84aa-kube-api-access-9tzwb\") pod \"opendatahub-operator-controller-manager-6755fff5c-lr4lb\" (UID: \"3f7374a7-4abb-48a2-9704-748bf75c84aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb"
Nov 02 01:37:43.643422 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.643425    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f7374a7-4abb-48a2-9704-748bf75c84aa-webhook-cert\") pod \"opendatahub-operator-controller-manager-6755fff5c-lr4lb\" (UID: \"3f7374a7-4abb-48a2-9704-748bf75c84aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb"
Nov 02 01:37:43.643678 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.643455    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3f7374a7-4abb-48a2-9704-748bf75c84aa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6755fff5c-lr4lb\" (UID: \"3f7374a7-4abb-48a2-9704-748bf75c84aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb"
Nov 02 01:37:43.643678 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.643495    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3f7374a7-4abb-48a2-9704-748bf75c84aa-metrics-cert\") pod \"opendatahub-operator-controller-manager-6755fff5c-lr4lb\" (UID: \"3f7374a7-4abb-48a2-9704-748bf75c84aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb"
Nov 02 01:37:43.643678 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.643526    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f7374a7-4abb-48a2-9704-748bf75c84aa-tmp\") pod \"opendatahub-operator-controller-manager-6755fff5c-lr4lb\" (UID: \"3f7374a7-4abb-48a2-9704-748bf75c84aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb"
Nov 02 01:37:43.643678 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.643546    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3f7374a7-4abb-48a2-9704-748bf75c84aa-manifests\") pod \"opendatahub-operator-controller-manager-6755fff5c-lr4lb\" (UID: \"3f7374a7-4abb-48a2-9704-748bf75c84aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb"
Nov 02 01:37:43.644017 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.643994    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3f7374a7-4abb-48a2-9704-748bf75c84aa-manifests\") pod \"opendatahub-operator-controller-manager-6755fff5c-lr4lb\" (UID: \"3f7374a7-4abb-48a2-9704-748bf75c84aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb"
Nov 02 01:37:43.646047 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.646021    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3f7374a7-4abb-48a2-9704-748bf75c84aa-metrics-cert\") pod \"opendatahub-operator-controller-manager-6755fff5c-lr4lb\" (UID: \"3f7374a7-4abb-48a2-9704-748bf75c84aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb"
Nov 02 01:37:43.646406 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.646384    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f7374a7-4abb-48a2-9704-748bf75c84aa-webhook-cert\") pod \"opendatahub-operator-controller-manager-6755fff5c-lr4lb\" (UID: \"3f7374a7-4abb-48a2-9704-748bf75c84aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb"
Nov 02 01:37:43.646574 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.646547    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3f7374a7-4abb-48a2-9704-748bf75c84aa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6755fff5c-lr4lb\" (UID: \"3f7374a7-4abb-48a2-9704-748bf75c84aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb"
Nov 02 01:37:43.646664 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.646646    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f7374a7-4abb-48a2-9704-748bf75c84aa-tmp\") pod \"opendatahub-operator-controller-manager-6755fff5c-lr4lb\" (UID: \"3f7374a7-4abb-48a2-9704-748bf75c84aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb"
Nov 02 01:37:43.704860 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.704839    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tzwb\" (UniqueName: \"kubernetes.io/projected/3f7374a7-4abb-48a2-9704-748bf75c84aa-kube-api-access-9tzwb\") pod \"opendatahub-operator-controller-manager-6755fff5c-lr4lb\" (UID: \"3f7374a7-4abb-48a2-9704-748bf75c84aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb"
Nov 02 01:37:43.760614 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:43.760587    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb"
Nov 02 01:37:44.247885 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:44.247846    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb"]
Nov 02 01:37:44.254864 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:37:44.254836    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7374a7_4abb_48a2_9704_748bf75c84aa.slice/crio-e754e8ac3bfbbbdcba94452fadf6398ebe13fcbde087a11187820389fd591228 WatchSource:0}: Error finding container e754e8ac3bfbbbdcba94452fadf6398ebe13fcbde087a11187820389fd591228: Status 404 returned error can't find the container with id e754e8ac3bfbbbdcba94452fadf6398ebe13fcbde087a11187820389fd591228
Nov 02 01:37:44.684568 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:44.684497    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-sfvtj"]
Nov 02 01:37:44.687048 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:44.687028    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sfvtj"
Nov 02 01:37:44.745260 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:44.745194    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 01:37:44.745260 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:44.745199    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 02 01:37:44.745446 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:44.745268    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 01:37:44.745446 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:44.745337    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-l47xb\""
Nov 02 01:37:44.745446 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:44.745344    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 02 01:37:44.745881 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:44.745846    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-sfvtj"]
Nov 02 01:37:44.756724 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:44.756697    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58bb5\" (UniqueName: \"kubernetes.io/projected/8c9b0d42-b9de-4e90-a03b-dd3b73e8e209-kube-api-access-58bb5\") pod \"kuadrant-console-plugin-66c4fb575d-sfvtj\" (UID: \"8c9b0d42-b9de-4e90-a03b-dd3b73e8e209\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sfvtj"
Nov 02 01:37:44.756838 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:44.756787    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c9b0d42-b9de-4e90-a03b-dd3b73e8e209-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-sfvtj\" (UID: \"8c9b0d42-b9de-4e90-a03b-dd3b73e8e209\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sfvtj"
Nov 02 01:37:44.756884 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:44.756845    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8c9b0d42-b9de-4e90-a03b-dd3b73e8e209-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-sfvtj\" (UID: \"8c9b0d42-b9de-4e90-a03b-dd3b73e8e209\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sfvtj"
Nov 02 01:37:44.857746 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:44.857721    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8c9b0d42-b9de-4e90-a03b-dd3b73e8e209-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-sfvtj\" (UID: \"8c9b0d42-b9de-4e90-a03b-dd3b73e8e209\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sfvtj"
Nov 02 01:37:44.857848 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:44.857785    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58bb5\" (UniqueName: \"kubernetes.io/projected/8c9b0d42-b9de-4e90-a03b-dd3b73e8e209-kube-api-access-58bb5\") pod \"kuadrant-console-plugin-66c4fb575d-sfvtj\" (UID: \"8c9b0d42-b9de-4e90-a03b-dd3b73e8e209\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sfvtj"
Nov 02 01:37:44.857848 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:44.857835    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c9b0d42-b9de-4e90-a03b-dd3b73e8e209-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-sfvtj\" (UID: \"8c9b0d42-b9de-4e90-a03b-dd3b73e8e209\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sfvtj"
Nov 02 01:37:44.858452 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:44.858436    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8c9b0d42-b9de-4e90-a03b-dd3b73e8e209-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-sfvtj\" (UID: \"8c9b0d42-b9de-4e90-a03b-dd3b73e8e209\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sfvtj"
Nov 02 01:37:44.860181 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:44.860163    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c9b0d42-b9de-4e90-a03b-dd3b73e8e209-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-sfvtj\" (UID: \"8c9b0d42-b9de-4e90-a03b-dd3b73e8e209\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sfvtj"
Nov 02 01:37:44.918849 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:44.918821    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58bb5\" (UniqueName: \"kubernetes.io/projected/8c9b0d42-b9de-4e90-a03b-dd3b73e8e209-kube-api-access-58bb5\") pod \"kuadrant-console-plugin-66c4fb575d-sfvtj\" (UID: \"8c9b0d42-b9de-4e90-a03b-dd3b73e8e209\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sfvtj"
Nov 02 01:37:44.997449 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:44.997431    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sfvtj"
Nov 02 01:37:44.998742 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:44.998716    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb" event={"ID":"3f7374a7-4abb-48a2-9704-748bf75c84aa","Type":"ContainerStarted","Data":"e754e8ac3bfbbbdcba94452fadf6398ebe13fcbde087a11187820389fd591228"}
Nov 02 01:37:45.380933 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:45.380865    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-sfvtj"]
Nov 02 01:37:45.383962 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:37:45.383931    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c9b0d42_b9de_4e90_a03b_dd3b73e8e209.slice/crio-0f6b9f4339f71553c2eed18709657dab67e66aa7bbcf2531366d13dc346ffa3a WatchSource:0}: Error finding container 0f6b9f4339f71553c2eed18709657dab67e66aa7bbcf2531366d13dc346ffa3a: Status 404 returned error can't find the container with id 0f6b9f4339f71553c2eed18709657dab67e66aa7bbcf2531366d13dc346ffa3a
Nov 02 01:37:46.006346 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:46.006310    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sfvtj" event={"ID":"8c9b0d42-b9de-4e90-a03b-dd3b73e8e209","Type":"ContainerStarted","Data":"0f6b9f4339f71553c2eed18709657dab67e66aa7bbcf2531366d13dc346ffa3a"}
Nov 02 01:37:48.015817 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:48.015742    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb" event={"ID":"3f7374a7-4abb-48a2-9704-748bf75c84aa","Type":"ContainerStarted","Data":"93cb616f568357b80b45da72875385ac0c1a8d9567efe9d98cb91fd907b16e16"}
Nov 02 01:37:49.022140 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:49.022109    5875 generic.go:356] "Generic (PLEG): container finished" podID="3f7374a7-4abb-48a2-9704-748bf75c84aa" containerID="93cb616f568357b80b45da72875385ac0c1a8d9567efe9d98cb91fd907b16e16" exitCode=0
Nov 02 01:37:49.022637 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:49.022189    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb" event={"ID":"3f7374a7-4abb-48a2-9704-748bf75c84aa","Type":"ContainerDied","Data":"93cb616f568357b80b45da72875385ac0c1a8d9567efe9d98cb91fd907b16e16"}
Nov 02 01:37:50.029324 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:50.029283    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb" event={"ID":"3f7374a7-4abb-48a2-9704-748bf75c84aa","Type":"ContainerStarted","Data":"f335127a57478da9a5d2a0e7aa49845ad2a9cf25ac253bc742b85fcf8e4f74f9"}
Nov 02 01:37:50.029785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:50.029378    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb"
Nov 02 01:37:50.295606 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:50.295547    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb" podStartSLOduration=2.50352285 podStartE2EDuration="7.295531755s" totalImagesPullingTime="4.294282452s" totalInitContainerRuntime="497.726453ms" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:50.246067887 +0000 UTC m=+317.687104387" watchObservedRunningTime="2026-11-02 01:37:50.295531755 +0000 UTC m=+317.736568252"
Nov 02 01:37:54.051379 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:54.051340    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sfvtj" event={"ID":"8c9b0d42-b9de-4e90-a03b-dd3b73e8e209","Type":"ContainerStarted","Data":"8d51d7f786a3e5ae70dc219ba4d5eda3c333473e43bcc27e7ecb862a94e1a642"}
Nov 02 01:37:54.168182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:54.168133    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sfvtj" podStartSLOduration=2.114856731 podStartE2EDuration="10.16811711s" totalImagesPullingTime="8.053260379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:54.166697827 +0000 UTC m=+321.607734325" watchObservedRunningTime="2026-11-02 01:37:54.16811711 +0000 UTC m=+321.609153607"
Nov 02 01:37:57.726759 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:57.726724    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-w4dlf"]
Nov 02 01:37:57.731433 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:57.731409    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4dlf"
Nov 02 01:37:57.783275 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:57.783253    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-85wx9\""
Nov 02 01:37:57.832836 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:57.832814    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 02 01:37:57.833039 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:57.833024    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 02 01:37:57.834110 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:57.834091    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-w4dlf"]
Nov 02 01:37:57.890648 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:57.890620    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6727k\" (UniqueName: \"kubernetes.io/projected/12377e32-6225-4e0c-b84a-e73531197f27-kube-api-access-6727k\") pod \"cert-manager-webhook-5dbbbf75cc-w4dlf\" (UID: \"12377e32-6225-4e0c-b84a-e73531197f27\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4dlf"
Nov 02 01:37:57.890744 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:57.890662    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12377e32-6225-4e0c-b84a-e73531197f27-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-w4dlf\" (UID: \"12377e32-6225-4e0c-b84a-e73531197f27\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4dlf"
Nov 02 01:37:57.991427 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:57.991356    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6727k\" (UniqueName: \"kubernetes.io/projected/12377e32-6225-4e0c-b84a-e73531197f27-kube-api-access-6727k\") pod \"cert-manager-webhook-5dbbbf75cc-w4dlf\" (UID: \"12377e32-6225-4e0c-b84a-e73531197f27\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4dlf"
Nov 02 01:37:57.991427 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:57.991390    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12377e32-6225-4e0c-b84a-e73531197f27-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-w4dlf\" (UID: \"12377e32-6225-4e0c-b84a-e73531197f27\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4dlf"
Nov 02 01:37:58.052134 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:58.052107    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12377e32-6225-4e0c-b84a-e73531197f27-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-w4dlf\" (UID: \"12377e32-6225-4e0c-b84a-e73531197f27\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4dlf"
Nov 02 01:37:58.052304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:58.052285    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6727k\" (UniqueName: \"kubernetes.io/projected/12377e32-6225-4e0c-b84a-e73531197f27-kube-api-access-6727k\") pod \"cert-manager-webhook-5dbbbf75cc-w4dlf\" (UID: \"12377e32-6225-4e0c-b84a-e73531197f27\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4dlf"
Nov 02 01:37:58.342795 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:58.342714    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4dlf"
Nov 02 01:37:58.713053 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:58.712986    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-w4dlf"]
Nov 02 01:37:58.715650 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:37:58.715625    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12377e32_6225_4e0c_b84a_e73531197f27.slice/crio-6be90af5aa4eb0949c871bcaf7d0eb0d1defd15ec43b91380fd04167be280f25 WatchSource:0}: Error finding container 6be90af5aa4eb0949c871bcaf7d0eb0d1defd15ec43b91380fd04167be280f25: Status 404 returned error can't find the container with id 6be90af5aa4eb0949c871bcaf7d0eb0d1defd15ec43b91380fd04167be280f25
Nov 02 01:37:59.073949 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:37:59.073866    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4dlf" event={"ID":"12377e32-6225-4e0c-b84a-e73531197f27","Type":"ContainerStarted","Data":"6be90af5aa4eb0949c871bcaf7d0eb0d1defd15ec43b91380fd04167be280f25"}
Nov 02 01:38:01.037390 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:01.037358    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6755fff5c-lr4lb"
Nov 02 01:38:04.099379 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:04.099344    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4dlf" event={"ID":"12377e32-6225-4e0c-b84a-e73531197f27","Type":"ContainerStarted","Data":"3519805487a116b822747e68687ebbc3cd734990f75e49889b267dc0ec7baedf"}
Nov 02 01:38:04.099769 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:04.099401    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4dlf"
Nov 02 01:38:04.216073 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:04.216021    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4dlf" podStartSLOduration=2.643758922 podStartE2EDuration="7.21600388s" totalImagesPullingTime="4.572244958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:04.21513819 +0000 UTC m=+331.656174701" watchObservedRunningTime="2026-11-02 01:38:04.21600388 +0000 UTC m=+331.657040374"
Nov 02 01:38:07.531367 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:07.531328    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lqzgg"]
Nov 02 01:38:07.535191 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:07.535167    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lqzgg"
Nov 02 01:38:07.574211 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:07.574185    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htcpp\" (UniqueName: \"kubernetes.io/projected/a29dfc50-61e7-4b88-9e66-cf4ff1938e44-kube-api-access-htcpp\") pod \"cert-manager-cainjector-5c57bbd4fc-lqzgg\" (UID: \"a29dfc50-61e7-4b88-9e66-cf4ff1938e44\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lqzgg"
Nov 02 01:38:07.574321 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:07.574265    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a29dfc50-61e7-4b88-9e66-cf4ff1938e44-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lqzgg\" (UID: \"a29dfc50-61e7-4b88-9e66-cf4ff1938e44\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lqzgg"
Nov 02 01:38:07.592065 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:07.592037    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-htlg9\""
Nov 02 01:38:07.639881 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:07.639856    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lqzgg"]
Nov 02 01:38:07.675065 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:07.675042    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htcpp\" (UniqueName: \"kubernetes.io/projected/a29dfc50-61e7-4b88-9e66-cf4ff1938e44-kube-api-access-htcpp\") pod \"cert-manager-cainjector-5c57bbd4fc-lqzgg\" (UID: \"a29dfc50-61e7-4b88-9e66-cf4ff1938e44\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lqzgg"
Nov 02 01:38:07.675163 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:07.675107    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a29dfc50-61e7-4b88-9e66-cf4ff1938e44-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lqzgg\" (UID: \"a29dfc50-61e7-4b88-9e66-cf4ff1938e44\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lqzgg"
Nov 02 01:38:07.735556 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:07.735532    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a29dfc50-61e7-4b88-9e66-cf4ff1938e44-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lqzgg\" (UID: \"a29dfc50-61e7-4b88-9e66-cf4ff1938e44\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lqzgg"
Nov 02 01:38:07.735674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:07.735652    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htcpp\" (UniqueName: \"kubernetes.io/projected/a29dfc50-61e7-4b88-9e66-cf4ff1938e44-kube-api-access-htcpp\") pod \"cert-manager-cainjector-5c57bbd4fc-lqzgg\" (UID: \"a29dfc50-61e7-4b88-9e66-cf4ff1938e44\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lqzgg"
Nov 02 01:38:07.846843 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:07.846784    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lqzgg"
Nov 02 01:38:08.218737 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:08.218705    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lqzgg"]
Nov 02 01:38:08.221574 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:38:08.221535    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda29dfc50_61e7_4b88_9e66_cf4ff1938e44.slice/crio-a6806303633116c04ff0e177ef391564213b15efbe395d03e193a9cf93251e9b WatchSource:0}: Error finding container a6806303633116c04ff0e177ef391564213b15efbe395d03e193a9cf93251e9b: Status 404 returned error can't find the container with id a6806303633116c04ff0e177ef391564213b15efbe395d03e193a9cf93251e9b
Nov 02 01:38:09.122299 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:09.122262    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lqzgg" event={"ID":"a29dfc50-61e7-4b88-9e66-cf4ff1938e44","Type":"ContainerStarted","Data":"2832d8455aa96953d4501eac57002d369ee44b5a80c294fd30f76cc1dc41c420"}
Nov 02 01:38:09.122793 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:09.122315    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lqzgg" event={"ID":"a29dfc50-61e7-4b88-9e66-cf4ff1938e44","Type":"ContainerStarted","Data":"a6806303633116c04ff0e177ef391564213b15efbe395d03e193a9cf93251e9b"}
Nov 02 01:38:09.237168 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:09.237119    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lqzgg" podStartSLOduration=2.23710433 podStartE2EDuration="2.23710433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:09.23635816 +0000 UTC m=+336.677394659" watchObservedRunningTime="2026-11-02 01:38:09.23710433 +0000 UTC m=+336.678140827"
Nov 02 01:38:10.105469 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:10.105442    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w4dlf"
Nov 02 01:38:11.319054 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:38:11.319016    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:38:14.448609 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:14.448578    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-bf99ddbdb-jkkrf"]
Nov 02 01:38:14.452527 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:14.452511    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-bf99ddbdb-jkkrf"
Nov 02 01:38:14.505574 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:14.505546    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 01:38:14.505688 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:14.505625    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 01:38:14.532299 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:14.532267    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hwfm\" (UniqueName: \"kubernetes.io/projected/8e0923dd-1994-424e-93c7-dd9cd1431dc5-kube-api-access-7hwfm\") pod \"kube-auth-proxy-bf99ddbdb-jkkrf\" (UID: \"8e0923dd-1994-424e-93c7-dd9cd1431dc5\") " pod="openshift-ingress/kube-auth-proxy-bf99ddbdb-jkkrf"
Nov 02 01:38:14.532415 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:14.532322    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e0923dd-1994-424e-93c7-dd9cd1431dc5-tls-certs\") pod \"kube-auth-proxy-bf99ddbdb-jkkrf\" (UID: \"8e0923dd-1994-424e-93c7-dd9cd1431dc5\") " pod="openshift-ingress/kube-auth-proxy-bf99ddbdb-jkkrf"
Nov 02 01:38:14.532465 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:14.532443    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e0923dd-1994-424e-93c7-dd9cd1431dc5-tmp\") pod \"kube-auth-proxy-bf99ddbdb-jkkrf\" (UID: \"8e0923dd-1994-424e-93c7-dd9cd1431dc5\") " pod="openshift-ingress/kube-auth-proxy-bf99ddbdb-jkkrf"
Nov 02 01:38:14.555084 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:14.555060    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 02 01:38:14.555084 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:14.555079    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 02 01:38:14.555302 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:14.555285    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9jznf\""
Nov 02 01:38:14.556473 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:14.556450    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-bf99ddbdb-jkkrf"]
Nov 02 01:38:14.633302 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:14.633271    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e0923dd-1994-424e-93c7-dd9cd1431dc5-tmp\") pod \"kube-auth-proxy-bf99ddbdb-jkkrf\" (UID: \"8e0923dd-1994-424e-93c7-dd9cd1431dc5\") " pod="openshift-ingress/kube-auth-proxy-bf99ddbdb-jkkrf"
Nov 02 01:38:14.633302 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:14.633306    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hwfm\" (UniqueName: \"kubernetes.io/projected/8e0923dd-1994-424e-93c7-dd9cd1431dc5-kube-api-access-7hwfm\") pod \"kube-auth-proxy-bf99ddbdb-jkkrf\" (UID: \"8e0923dd-1994-424e-93c7-dd9cd1431dc5\") " pod="openshift-ingress/kube-auth-proxy-bf99ddbdb-jkkrf"
Nov 02 01:38:14.633474 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:14.633441    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e0923dd-1994-424e-93c7-dd9cd1431dc5-tls-certs\") pod \"kube-auth-proxy-bf99ddbdb-jkkrf\" (UID: \"8e0923dd-1994-424e-93c7-dd9cd1431dc5\") " pod="openshift-ingress/kube-auth-proxy-bf99ddbdb-jkkrf"
Nov 02 01:38:14.635614 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:14.635591    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e0923dd-1994-424e-93c7-dd9cd1431dc5-tmp\") pod \"kube-auth-proxy-bf99ddbdb-jkkrf\" (UID: \"8e0923dd-1994-424e-93c7-dd9cd1431dc5\") " pod="openshift-ingress/kube-auth-proxy-bf99ddbdb-jkkrf"
Nov 02 01:38:14.635803 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:14.635790    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e0923dd-1994-424e-93c7-dd9cd1431dc5-tls-certs\") pod \"kube-auth-proxy-bf99ddbdb-jkkrf\" (UID: \"8e0923dd-1994-424e-93c7-dd9cd1431dc5\") " pod="openshift-ingress/kube-auth-proxy-bf99ddbdb-jkkrf"
Nov 02 01:38:14.693786 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:14.693764    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hwfm\" (UniqueName: \"kubernetes.io/projected/8e0923dd-1994-424e-93c7-dd9cd1431dc5-kube-api-access-7hwfm\") pod \"kube-auth-proxy-bf99ddbdb-jkkrf\" (UID: \"8e0923dd-1994-424e-93c7-dd9cd1431dc5\") " pod="openshift-ingress/kube-auth-proxy-bf99ddbdb-jkkrf"
Nov 02 01:38:14.763734 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:14.763714    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-bf99ddbdb-jkkrf"
Nov 02 01:38:15.134707 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:15.134495    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-bf99ddbdb-jkkrf"]
Nov 02 01:38:15.137183 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:38:15.137155    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0923dd_1994_424e_93c7_dd9cd1431dc5.slice/crio-f3a009153c0d2ed6f4fd9848004f2d21959e3d3bc941e5cf7f4b525b05c3035a WatchSource:0}: Error finding container f3a009153c0d2ed6f4fd9848004f2d21959e3d3bc941e5cf7f4b525b05c3035a: Status 404 returned error can't find the container with id f3a009153c0d2ed6f4fd9848004f2d21959e3d3bc941e5cf7f4b525b05c3035a
Nov 02 01:38:15.149393 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:15.149369    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-bf99ddbdb-jkkrf" event={"ID":"8e0923dd-1994-424e-93c7-dd9cd1431dc5","Type":"ContainerStarted","Data":"f3a009153c0d2ed6f4fd9848004f2d21959e3d3bc941e5cf7f4b525b05c3035a"}
Nov 02 01:38:19.170412 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:19.170357    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-bf99ddbdb-jkkrf" event={"ID":"8e0923dd-1994-424e-93c7-dd9cd1431dc5","Type":"ContainerStarted","Data":"be9f8e7dd3a74414100c359a582c8ff600a8f91fb7568b6ebcee8eb76d0e04b9"}
Nov 02 01:38:19.286304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:19.286253    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-bf99ddbdb-jkkrf" podStartSLOduration=1.341174447 podStartE2EDuration="5.286213748s" totalImagesPullingTime="3.945039301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:19.285033694 +0000 UTC m=+346.726070194" watchObservedRunningTime="2026-11-02 01:38:19.286213748 +0000 UTC m=+346.727250242"
Nov 02 01:38:27.233471 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:27.233438    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nlznl"]
Nov 02 01:38:27.250163 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:27.250138    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-nlznl"
Nov 02 01:38:27.294044 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:27.294015    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nlznl"]
Nov 02 01:38:27.342327 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:27.342296    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 02 01:38:27.344271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:27.344078    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/26da4a73-1310-4ac7-9987-738c6d5fc709-config-file\") pod \"limitador-limitador-74dfcfb894-nlznl\" (UID: \"26da4a73-1310-4ac7-9987-738c6d5fc709\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-nlznl"
Nov 02 01:38:27.344271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:27.344156    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmcqq\" (UniqueName: \"kubernetes.io/projected/26da4a73-1310-4ac7-9987-738c6d5fc709-kube-api-access-hmcqq\") pod \"limitador-limitador-74dfcfb894-nlznl\" (UID: \"26da4a73-1310-4ac7-9987-738c6d5fc709\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-nlznl"
Nov 02 01:38:27.445382 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:27.445351    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/26da4a73-1310-4ac7-9987-738c6d5fc709-config-file\") pod \"limitador-limitador-74dfcfb894-nlznl\" (UID: \"26da4a73-1310-4ac7-9987-738c6d5fc709\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-nlznl"
Nov 02 01:38:27.445570 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:27.445409    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmcqq\" (UniqueName: \"kubernetes.io/projected/26da4a73-1310-4ac7-9987-738c6d5fc709-kube-api-access-hmcqq\") pod \"limitador-limitador-74dfcfb894-nlznl\" (UID: \"26da4a73-1310-4ac7-9987-738c6d5fc709\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-nlznl"
Nov 02 01:38:27.445991 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:27.445970    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/26da4a73-1310-4ac7-9987-738c6d5fc709-config-file\") pod \"limitador-limitador-74dfcfb894-nlznl\" (UID: \"26da4a73-1310-4ac7-9987-738c6d5fc709\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-nlznl"
Nov 02 01:38:27.483947 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:27.483873    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nlznl"]
Nov 02 01:38:27.506450 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:27.506423    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmcqq\" (UniqueName: \"kubernetes.io/projected/26da4a73-1310-4ac7-9987-738c6d5fc709-kube-api-access-hmcqq\") pod \"limitador-limitador-74dfcfb894-nlznl\" (UID: \"26da4a73-1310-4ac7-9987-738c6d5fc709\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-nlznl"
Nov 02 01:38:27.563150 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:27.563116    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-nlznl"
Nov 02 01:38:27.933413 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:27.933389    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nlznl"]
Nov 02 01:38:27.936056 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:38:27.936028    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26da4a73_1310_4ac7_9987_738c6d5fc709.slice/crio-f95a0ca8f6da9ca3f11313ad381e694cfe2f4c0a167c06e179a617b9162f12b6 WatchSource:0}: Error finding container f95a0ca8f6da9ca3f11313ad381e694cfe2f4c0a167c06e179a617b9162f12b6: Status 404 returned error can't find the container with id f95a0ca8f6da9ca3f11313ad381e694cfe2f4c0a167c06e179a617b9162f12b6
Nov 02 01:38:28.215020 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:28.214982    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-nlznl" event={"ID":"26da4a73-1310-4ac7-9987-738c6d5fc709","Type":"ContainerStarted","Data":"f95a0ca8f6da9ca3f11313ad381e694cfe2f4c0a167c06e179a617b9162f12b6"}
Nov 02 01:38:30.414014 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:30.413975    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-z8lgq"]
Nov 02 01:38:30.417974 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:30.417950    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:38:30.418123 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:30.418072    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-z8lgq"
Nov 02 01:38:30.421595 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:30.421577    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"
Nov 02 01:38:30.472479 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:30.472454    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-799rl\""
Nov 02 01:38:30.472822 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:30.472802    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd5qg\" (UniqueName: \"kubernetes.io/projected/f1f4adf7-97b0-4b57-a794-dfb041751d12-kube-api-access-cd5qg\") pod \"authorino-56c6c9ddd5-z8lgq\" (UID: \"f1f4adf7-97b0-4b57-a794-dfb041751d12\") " pod="kuadrant-system/authorino-56c6c9ddd5-z8lgq"
Nov 02 01:38:30.472911 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:30.472852    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2bfb1fba-9189-4c25-9805-ebe4bc75da01-config-file\") pod \"limitador-limitador-7c9b87d574-xgf4f\" (UID: \"2bfb1fba-9189-4c25-9805-ebe4bc75da01\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"
Nov 02 01:38:30.473016 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:30.472955    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kgj7\" (UniqueName: \"kubernetes.io/projected/2bfb1fba-9189-4c25-9805-ebe4bc75da01-kube-api-access-6kgj7\") pod \"limitador-limitador-7c9b87d574-xgf4f\" (UID: \"2bfb1fba-9189-4c25-9805-ebe4bc75da01\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"
Nov 02 01:38:30.474104 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:30.474075    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-z8lgq"]
Nov 02 01:38:30.523193 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:30.523158    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:38:30.574161 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:30.574126    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd5qg\" (UniqueName: \"kubernetes.io/projected/f1f4adf7-97b0-4b57-a794-dfb041751d12-kube-api-access-cd5qg\") pod \"authorino-56c6c9ddd5-z8lgq\" (UID: \"f1f4adf7-97b0-4b57-a794-dfb041751d12\") " pod="kuadrant-system/authorino-56c6c9ddd5-z8lgq"
Nov 02 01:38:30.574317 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:30.574174    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2bfb1fba-9189-4c25-9805-ebe4bc75da01-config-file\") pod \"limitador-limitador-7c9b87d574-xgf4f\" (UID: \"2bfb1fba-9189-4c25-9805-ebe4bc75da01\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"
Nov 02 01:38:30.574317 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:30.574204    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kgj7\" (UniqueName: \"kubernetes.io/projected/2bfb1fba-9189-4c25-9805-ebe4bc75da01-kube-api-access-6kgj7\") pod \"limitador-limitador-7c9b87d574-xgf4f\" (UID: \"2bfb1fba-9189-4c25-9805-ebe4bc75da01\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"
Nov 02 01:38:30.575041 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:30.575022    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2bfb1fba-9189-4c25-9805-ebe4bc75da01-config-file\") pod \"limitador-limitador-7c9b87d574-xgf4f\" (UID: \"2bfb1fba-9189-4c25-9805-ebe4bc75da01\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"
Nov 02 01:38:30.635702 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:30.635677    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kgj7\" (UniqueName: \"kubernetes.io/projected/2bfb1fba-9189-4c25-9805-ebe4bc75da01-kube-api-access-6kgj7\") pod \"limitador-limitador-7c9b87d574-xgf4f\" (UID: \"2bfb1fba-9189-4c25-9805-ebe4bc75da01\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"
Nov 02 01:38:30.635838 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:30.635725    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd5qg\" (UniqueName: \"kubernetes.io/projected/f1f4adf7-97b0-4b57-a794-dfb041751d12-kube-api-access-cd5qg\") pod \"authorino-56c6c9ddd5-z8lgq\" (UID: \"f1f4adf7-97b0-4b57-a794-dfb041751d12\") " pod="kuadrant-system/authorino-56c6c9ddd5-z8lgq"
Nov 02 01:38:30.734847 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:30.734818    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-z8lgq"
Nov 02 01:38:30.743242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:30.743201    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"
Nov 02 01:38:30.744397 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:30.744313    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:38:31.114324 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:31.114252    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-z8lgq"]
Nov 02 01:38:31.117659 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:38:31.117629    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1f4adf7_97b0_4b57_a794_dfb041751d12.slice/crio-fe51ae315285df6a758ea80695c6290fab54e9f73385ba9864d71f1e62185a6f WatchSource:0}: Error finding container fe51ae315285df6a758ea80695c6290fab54e9f73385ba9864d71f1e62185a6f: Status 404 returned error can't find the container with id fe51ae315285df6a758ea80695c6290fab54e9f73385ba9864d71f1e62185a6f
Nov 02 01:38:31.165582 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:31.165556    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:38:31.169027 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:38:31.168996    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bfb1fba_9189_4c25_9805_ebe4bc75da01.slice/crio-9c9a8cdabcab265950c7693564de7f9b0e3a3f6d446e41c3629b5574bdeb0d14 WatchSource:0}: Error finding container 9c9a8cdabcab265950c7693564de7f9b0e3a3f6d446e41c3629b5574bdeb0d14: Status 404 returned error can't find the container with id 9c9a8cdabcab265950c7693564de7f9b0e3a3f6d446e41c3629b5574bdeb0d14
Nov 02 01:38:31.231613 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:31.231492    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-z8lgq" event={"ID":"f1f4adf7-97b0-4b57-a794-dfb041751d12","Type":"ContainerStarted","Data":"fe51ae315285df6a758ea80695c6290fab54e9f73385ba9864d71f1e62185a6f"}
Nov 02 01:38:31.232827 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:31.232796    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-xgf4f" event={"ID":"2bfb1fba-9189-4c25-9805-ebe4bc75da01","Type":"ContainerStarted","Data":"9c9a8cdabcab265950c7693564de7f9b0e3a3f6d446e41c3629b5574bdeb0d14"}
Nov 02 01:38:32.244966 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:32.244924    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-nlznl" event={"ID":"26da4a73-1310-4ac7-9987-738c6d5fc709","Type":"ContainerStarted","Data":"59d02a83da80d4db0c0e24840312d26307668961b651cdf5fe8778533ef47d53"}
Nov 02 01:38:32.245442 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:32.245038    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-nlznl"
Nov 02 01:38:32.359590 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:32.359539    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-nlznl" podStartSLOduration=1.5933143699999999 podStartE2EDuration="5.359521244s" totalImagesPullingTime="3.766206874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:32.359022133 +0000 UTC m=+359.800058635" watchObservedRunningTime="2026-11-02 01:38:32.359521244 +0000 UTC m=+359.800557742"
Nov 02 01:38:33.251103 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:33.251070    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-xgf4f" event={"ID":"2bfb1fba-9189-4c25-9805-ebe4bc75da01","Type":"ContainerStarted","Data":"47cb7b5fe162468d9b7af698dc3ba9f142aea397666b64cf07ae53cf831663b4"}
Nov 02 01:38:33.251516 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:33.251356    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"
Nov 02 01:38:33.365835 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:33.365789    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-xgf4f" podStartSLOduration=2.180724456 podStartE2EDuration="3.365774809s" totalImagesPullingTime="1.185050353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:33.364896718 +0000 UTC m=+360.805933242" watchObservedRunningTime="2026-11-02 01:38:33.365774809 +0000 UTC m=+360.806811284"
Nov 02 01:38:35.260617 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:35.260572    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-z8lgq" event={"ID":"f1f4adf7-97b0-4b57-a794-dfb041751d12","Type":"ContainerStarted","Data":"19e311e4494a69339c66b721add91dc681859ae3b412cffb09aee354dcdb5ab3"}
Nov 02 01:38:35.374828 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:35.374764    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-z8lgq" podStartSLOduration=1.7671899610000001 podStartE2EDuration="5.374744049s" totalImagesPullingTime="3.607554088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:35.373271854 +0000 UTC m=+362.814308352" watchObservedRunningTime="2026-11-02 01:38:35.374744049 +0000 UTC m=+362.815780548"
Nov 02 01:38:41.317977 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:38:41.317896    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:38:43.252091 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:43.252063    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-nlznl"
Nov 02 01:38:44.256411 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:44.256380    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"
Nov 02 01:38:44.422665 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:44.422625    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nlznl"]
Nov 02 01:38:44.422875 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:44.422849    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-nlznl" podUID="26da4a73-1310-4ac7-9987-738c6d5fc709" containerName="limitador" containerID="cri-o://59d02a83da80d4db0c0e24840312d26307668961b651cdf5fe8778533ef47d53" gracePeriod=30
Nov 02 01:38:44.970113 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:44.970091    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-nlznl"
Nov 02 01:38:45.109989 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:45.109916    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26da4a73-1310-4ac7-9987-738c6d5fc709-kube-api-access-hmcqq\" (UniqueName: \"kubernetes.io/projected/26da4a73-1310-4ac7-9987-738c6d5fc709-kube-api-access-hmcqq\") pod \"26da4a73-1310-4ac7-9987-738c6d5fc709\" (UID: \"26da4a73-1310-4ac7-9987-738c6d5fc709\") "
Nov 02 01:38:45.110131 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:45.109997    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26da4a73-1310-4ac7-9987-738c6d5fc709-config-file\" (UniqueName: \"kubernetes.io/configmap/26da4a73-1310-4ac7-9987-738c6d5fc709-config-file\") pod \"26da4a73-1310-4ac7-9987-738c6d5fc709\" (UID: \"26da4a73-1310-4ac7-9987-738c6d5fc709\") "
Nov 02 01:38:45.110365 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:45.110346    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26da4a73-1310-4ac7-9987-738c6d5fc709-config-file" pod "26da4a73-1310-4ac7-9987-738c6d5fc709" (UID: "26da4a73-1310-4ac7-9987-738c6d5fc709"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:38:45.112180 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:45.112161    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26da4a73-1310-4ac7-9987-738c6d5fc709-kube-api-access-hmcqq" pod "26da4a73-1310-4ac7-9987-738c6d5fc709" (UID: "26da4a73-1310-4ac7-9987-738c6d5fc709"). InnerVolumeSpecName "kube-api-access-hmcqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:38:45.210847 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:45.210811    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmcqq\" (UniqueName: \"kubernetes.io/projected/26da4a73-1310-4ac7-9987-738c6d5fc709-kube-api-access-hmcqq\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:38:45.210847 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:45.210843    5875 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/26da4a73-1310-4ac7-9987-738c6d5fc709-config-file\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:38:45.304108 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:45.304070    5875 generic.go:356] "Generic (PLEG): container finished" podID="26da4a73-1310-4ac7-9987-738c6d5fc709" containerID="59d02a83da80d4db0c0e24840312d26307668961b651cdf5fe8778533ef47d53" exitCode=0
Nov 02 01:38:45.304549 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:45.304118    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-nlznl"
Nov 02 01:38:45.304549 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:45.304146    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-nlznl" event={"ID":"26da4a73-1310-4ac7-9987-738c6d5fc709","Type":"ContainerDied","Data":"59d02a83da80d4db0c0e24840312d26307668961b651cdf5fe8778533ef47d53"}
Nov 02 01:38:45.304549 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:45.304202    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-nlznl" event={"ID":"26da4a73-1310-4ac7-9987-738c6d5fc709","Type":"ContainerDied","Data":"f95a0ca8f6da9ca3f11313ad381e694cfe2f4c0a167c06e179a617b9162f12b6"}
Nov 02 01:38:45.304549 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:45.304247    5875 scope.go:122] "RemoveContainer" containerID="59d02a83da80d4db0c0e24840312d26307668961b651cdf5fe8778533ef47d53"
Nov 02 01:38:45.313271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:45.313257    5875 scope.go:122] "RemoveContainer" containerID="59d02a83da80d4db0c0e24840312d26307668961b651cdf5fe8778533ef47d53"
Nov 02 01:38:45.313535 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:38:45.313517    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59d02a83da80d4db0c0e24840312d26307668961b651cdf5fe8778533ef47d53\": container with ID starting with 59d02a83da80d4db0c0e24840312d26307668961b651cdf5fe8778533ef47d53 not found: ID does not exist" containerID="59d02a83da80d4db0c0e24840312d26307668961b651cdf5fe8778533ef47d53"
Nov 02 01:38:45.313597 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:45.313543    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59d02a83da80d4db0c0e24840312d26307668961b651cdf5fe8778533ef47d53"} err="failed to get container status \"59d02a83da80d4db0c0e24840312d26307668961b651cdf5fe8778533ef47d53\": rpc error: code = NotFound desc = could not find container \"59d02a83da80d4db0c0e24840312d26307668961b651cdf5fe8778533ef47d53\": container with ID starting with 59d02a83da80d4db0c0e24840312d26307668961b651cdf5fe8778533ef47d53 not found: ID does not exist"
Nov 02 01:38:45.468432 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:45.468404    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nlznl"]
Nov 02 01:38:45.517663 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:45.517632    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nlznl"]
Nov 02 01:38:46.692863 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:38:46.692830    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26da4a73-1310-4ac7-9987-738c6d5fc709" path="/var/lib/kubelet/pods/26da4a73-1310-4ac7-9987-738c6d5fc709/volumes"
Nov 02 01:39:11.318999 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:39:11.318961    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:39:12.001432 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.001398    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-wks74"]
Nov 02 01:39:12.001815 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.001802    5875 cpu_manager.go:414] "containerMap: removing container" podUID="26da4a73-1310-4ac7-9987-738c6d5fc709" containerName="limitador"
Nov 02 01:39:12.001815 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.001816    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26da4a73-1310-4ac7-9987-738c6d5fc709" containerName="limitador"
Nov 02 01:39:12.001910 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.001897    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="26da4a73-1310-4ac7-9987-738c6d5fc709" containerName="limitador"
Nov 02 01:39:12.004827 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.004808    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-wks74"
Nov 02 01:39:12.037780 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.037753    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de494104-8e40-42d5-a2c1-aafc35402300-webhook-cert\") pod \"maas-controller-579cf7c6d-wks74\" (UID: \"de494104-8e40-42d5-a2c1-aafc35402300\") " pod="opendatahub/maas-controller-579cf7c6d-wks74"
Nov 02 01:39:12.037978 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.037786    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de494104-8e40-42d5-a2c1-aafc35402300-metrics-certs\") pod \"maas-controller-579cf7c6d-wks74\" (UID: \"de494104-8e40-42d5-a2c1-aafc35402300\") " pod="opendatahub/maas-controller-579cf7c6d-wks74"
Nov 02 01:39:12.037978 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.037806    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgqwb\" (UniqueName: \"kubernetes.io/projected/de494104-8e40-42d5-a2c1-aafc35402300-kube-api-access-zgqwb\") pod \"maas-controller-579cf7c6d-wks74\" (UID: \"de494104-8e40-42d5-a2c1-aafc35402300\") " pod="opendatahub/maas-controller-579cf7c6d-wks74"
Nov 02 01:39:12.056750 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.056722    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 01:39:12.056819 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.056726    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vj2vs\""
Nov 02 01:39:12.106167 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.106139    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 01:39:12.106167 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.106141    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 01:39:12.106353 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.106182    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 01:39:12.106353 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.106243    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 01:39:12.107552 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.107537    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-wks74"]
Nov 02 01:39:12.138671 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.138646    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de494104-8e40-42d5-a2c1-aafc35402300-webhook-cert\") pod \"maas-controller-579cf7c6d-wks74\" (UID: \"de494104-8e40-42d5-a2c1-aafc35402300\") " pod="opendatahub/maas-controller-579cf7c6d-wks74"
Nov 02 01:39:12.138820 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.138679    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de494104-8e40-42d5-a2c1-aafc35402300-metrics-certs\") pod \"maas-controller-579cf7c6d-wks74\" (UID: \"de494104-8e40-42d5-a2c1-aafc35402300\") " pod="opendatahub/maas-controller-579cf7c6d-wks74"
Nov 02 01:39:12.138820 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.138700    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgqwb\" (UniqueName: \"kubernetes.io/projected/de494104-8e40-42d5-a2c1-aafc35402300-kube-api-access-zgqwb\") pod \"maas-controller-579cf7c6d-wks74\" (UID: \"de494104-8e40-42d5-a2c1-aafc35402300\") " pod="opendatahub/maas-controller-579cf7c6d-wks74"
Nov 02 01:39:12.141275 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.141255    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de494104-8e40-42d5-a2c1-aafc35402300-metrics-certs\") pod \"maas-controller-579cf7c6d-wks74\" (UID: \"de494104-8e40-42d5-a2c1-aafc35402300\") " pod="opendatahub/maas-controller-579cf7c6d-wks74"
Nov 02 01:39:12.141370 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.141298    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de494104-8e40-42d5-a2c1-aafc35402300-webhook-cert\") pod \"maas-controller-579cf7c6d-wks74\" (UID: \"de494104-8e40-42d5-a2c1-aafc35402300\") " pod="opendatahub/maas-controller-579cf7c6d-wks74"
Nov 02 01:39:12.199339 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.199310    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgqwb\" (UniqueName: \"kubernetes.io/projected/de494104-8e40-42d5-a2c1-aafc35402300-kube-api-access-zgqwb\") pod \"maas-controller-579cf7c6d-wks74\" (UID: \"de494104-8e40-42d5-a2c1-aafc35402300\") " pod="opendatahub/maas-controller-579cf7c6d-wks74"
Nov 02 01:39:12.318454 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.318366    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-wks74"
Nov 02 01:39:12.422268 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.422237    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5f8c46b7cf-qn58p"]
Nov 02 01:39:12.426904 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.426876    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f8c46b7cf-qn58p"
Nov 02 01:39:12.484507 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.484474    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f8c46b7cf-qn58p"]
Nov 02 01:39:12.542277 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.542244    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tql7t\" (UniqueName: \"kubernetes.io/projected/f03055d8-42b7-4c1b-9900-4c891e0ac145-kube-api-access-tql7t\") pod \"maas-controller-5f8c46b7cf-qn58p\" (UID: \"f03055d8-42b7-4c1b-9900-4c891e0ac145\") " pod="opendatahub/maas-controller-5f8c46b7cf-qn58p"
Nov 02 01:39:12.542461 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.542346    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f03055d8-42b7-4c1b-9900-4c891e0ac145-webhook-cert\") pod \"maas-controller-5f8c46b7cf-qn58p\" (UID: \"f03055d8-42b7-4c1b-9900-4c891e0ac145\") " pod="opendatahub/maas-controller-5f8c46b7cf-qn58p"
Nov 02 01:39:12.542461 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.542395    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f03055d8-42b7-4c1b-9900-4c891e0ac145-metrics-certs\") pod \"maas-controller-5f8c46b7cf-qn58p\" (UID: \"f03055d8-42b7-4c1b-9900-4c891e0ac145\") " pod="opendatahub/maas-controller-5f8c46b7cf-qn58p"
Nov 02 01:39:12.642856 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.642788    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tql7t\" (UniqueName: \"kubernetes.io/projected/f03055d8-42b7-4c1b-9900-4c891e0ac145-kube-api-access-tql7t\") pod \"maas-controller-5f8c46b7cf-qn58p\" (UID: \"f03055d8-42b7-4c1b-9900-4c891e0ac145\") " pod="opendatahub/maas-controller-5f8c46b7cf-qn58p"
Nov 02 01:39:12.642856 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.642831    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f03055d8-42b7-4c1b-9900-4c891e0ac145-webhook-cert\") pod \"maas-controller-5f8c46b7cf-qn58p\" (UID: \"f03055d8-42b7-4c1b-9900-4c891e0ac145\") " pod="opendatahub/maas-controller-5f8c46b7cf-qn58p"
Nov 02 01:39:12.643076 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.642862    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f03055d8-42b7-4c1b-9900-4c891e0ac145-metrics-certs\") pod \"maas-controller-5f8c46b7cf-qn58p\" (UID: \"f03055d8-42b7-4c1b-9900-4c891e0ac145\") " pod="opendatahub/maas-controller-5f8c46b7cf-qn58p"
Nov 02 01:39:12.645355 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.645331    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f03055d8-42b7-4c1b-9900-4c891e0ac145-webhook-cert\") pod \"maas-controller-5f8c46b7cf-qn58p\" (UID: \"f03055d8-42b7-4c1b-9900-4c891e0ac145\") " pod="opendatahub/maas-controller-5f8c46b7cf-qn58p"
Nov 02 01:39:12.645458 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.645379    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f03055d8-42b7-4c1b-9900-4c891e0ac145-metrics-certs\") pod \"maas-controller-5f8c46b7cf-qn58p\" (UID: \"f03055d8-42b7-4c1b-9900-4c891e0ac145\") " pod="opendatahub/maas-controller-5f8c46b7cf-qn58p"
Nov 02 01:39:12.691786 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:39:12.691761    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde494104_8e40_42d5_a2c1_aafc35402300.slice/crio-c800b1bfd726a571c0272d55e87cbfbd994e95319340ce34cf483d237b05b519 WatchSource:0}: Error finding container c800b1bfd726a571c0272d55e87cbfbd994e95319340ce34cf483d237b05b519: Status 404 returned error can't find the container with id c800b1bfd726a571c0272d55e87cbfbd994e95319340ce34cf483d237b05b519
Nov 02 01:39:12.693291 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.693273    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-wks74"]
Nov 02 01:39:12.703363 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.703343    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tql7t\" (UniqueName: \"kubernetes.io/projected/f03055d8-42b7-4c1b-9900-4c891e0ac145-kube-api-access-tql7t\") pod \"maas-controller-5f8c46b7cf-qn58p\" (UID: \"f03055d8-42b7-4c1b-9900-4c891e0ac145\") " pod="opendatahub/maas-controller-5f8c46b7cf-qn58p"
Nov 02 01:39:12.739310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:12.739285    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f8c46b7cf-qn58p"
Nov 02 01:39:13.110090 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:13.110064    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f8c46b7cf-qn58p"]
Nov 02 01:39:13.113279 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:39:13.113248    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03055d8_42b7_4c1b_9900_4c891e0ac145.slice/crio-a6dfef63aad857793530ae12265990fe06de8edaecc9383161acb2f5c75aa485 WatchSource:0}: Error finding container a6dfef63aad857793530ae12265990fe06de8edaecc9383161acb2f5c75aa485: Status 404 returned error can't find the container with id a6dfef63aad857793530ae12265990fe06de8edaecc9383161acb2f5c75aa485
Nov 02 01:39:13.434702 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:13.434619    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-wks74" event={"ID":"de494104-8e40-42d5-a2c1-aafc35402300","Type":"ContainerStarted","Data":"c800b1bfd726a571c0272d55e87cbfbd994e95319340ce34cf483d237b05b519"}
Nov 02 01:39:13.435797 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:13.435768    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f8c46b7cf-qn58p" event={"ID":"f03055d8-42b7-4c1b-9900-4c891e0ac145","Type":"ContainerStarted","Data":"a6dfef63aad857793530ae12265990fe06de8edaecc9383161acb2f5c75aa485"}
Nov 02 01:39:16.455152 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:16.455117    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f8c46b7cf-qn58p" event={"ID":"f03055d8-42b7-4c1b-9900-4c891e0ac145","Type":"ContainerStarted","Data":"c4f88c6047525a6e8e8ab16f42258af2f67077aa2fdcd3ee4cc31fcee6c5f288"}
Nov 02 01:39:16.455649 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:16.455271    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5f8c46b7cf-qn58p"
Nov 02 01:39:16.457402 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:16.457378    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-wks74" event={"ID":"de494104-8e40-42d5-a2c1-aafc35402300","Type":"ContainerStarted","Data":"255b9fa428fc6bf0e57985714327b234be69aa03eea9dc8a703ff3db0b7b1c66"}
Nov 02 01:39:16.457547 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:16.457429    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-wks74"
Nov 02 01:39:16.603397 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:16.603342    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5f8c46b7cf-qn58p" podStartSLOduration=1.5194853579999998 podStartE2EDuration="4.603326064s" totalImagesPullingTime="3.083840706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:39:16.601821224 +0000 UTC m=+404.042857744" watchObservedRunningTime="2026-11-02 01:39:16.603326064 +0000 UTC m=+404.044362555"
Nov 02 01:39:16.718292 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:16.718171    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-wks74" podStartSLOduration=2.253867965 podStartE2EDuration="5.718153435s" totalImagesPullingTime="3.46428547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:39:16.717760964 +0000 UTC m=+404.158797463" watchObservedRunningTime="2026-11-02 01:39:16.718153435 +0000 UTC m=+404.159189930"
Nov 02 01:39:27.474066 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:27.474031    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5f8c46b7cf-qn58p"
Nov 02 01:39:27.474490 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:27.474123    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-wks74"
Nov 02 01:39:27.643312 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:27.643277    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-wks74"]
Nov 02 01:39:27.643603 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:27.643545    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-wks74" podUID="de494104-8e40-42d5-a2c1-aafc35402300" containerName="manager" containerID="cri-o://255b9fa428fc6bf0e57985714327b234be69aa03eea9dc8a703ff3db0b7b1c66" gracePeriod=30
Nov 02 01:39:27.887819 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:27.887795    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-wks74"
Nov 02 01:39:27.993337 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:27.993290    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de494104-8e40-42d5-a2c1-aafc35402300-webhook-cert\" (UniqueName: \"kubernetes.io/secret/de494104-8e40-42d5-a2c1-aafc35402300-webhook-cert\") pod \"de494104-8e40-42d5-a2c1-aafc35402300\" (UID: \"de494104-8e40-42d5-a2c1-aafc35402300\") "
Nov 02 01:39:27.993531 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:27.993389    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de494104-8e40-42d5-a2c1-aafc35402300-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de494104-8e40-42d5-a2c1-aafc35402300-metrics-certs\") pod \"de494104-8e40-42d5-a2c1-aafc35402300\" (UID: \"de494104-8e40-42d5-a2c1-aafc35402300\") "
Nov 02 01:39:27.993531 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:27.993458    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de494104-8e40-42d5-a2c1-aafc35402300-kube-api-access-zgqwb\" (UniqueName: \"kubernetes.io/projected/de494104-8e40-42d5-a2c1-aafc35402300-kube-api-access-zgqwb\") pod \"de494104-8e40-42d5-a2c1-aafc35402300\" (UID: \"de494104-8e40-42d5-a2c1-aafc35402300\") "
Nov 02 01:39:27.995937 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:27.995903    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de494104-8e40-42d5-a2c1-aafc35402300-metrics-certs" pod "de494104-8e40-42d5-a2c1-aafc35402300" (UID: "de494104-8e40-42d5-a2c1-aafc35402300"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:39:27.995937 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:27.995903    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de494104-8e40-42d5-a2c1-aafc35402300-kube-api-access-zgqwb" pod "de494104-8e40-42d5-a2c1-aafc35402300" (UID: "de494104-8e40-42d5-a2c1-aafc35402300"). InnerVolumeSpecName "kube-api-access-zgqwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:39:27.996087 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:27.995947    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de494104-8e40-42d5-a2c1-aafc35402300-webhook-cert" pod "de494104-8e40-42d5-a2c1-aafc35402300" (UID: "de494104-8e40-42d5-a2c1-aafc35402300"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:39:28.094766 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.094681    5875 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de494104-8e40-42d5-a2c1-aafc35402300-webhook-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:39:28.094766 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.094711    5875 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de494104-8e40-42d5-a2c1-aafc35402300-metrics-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:39:28.094766 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.094722    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgqwb\" (UniqueName: \"kubernetes.io/projected/de494104-8e40-42d5-a2c1-aafc35402300-kube-api-access-zgqwb\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:39:28.525440 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.525408    5875 generic.go:356] "Generic (PLEG): container finished" podID="de494104-8e40-42d5-a2c1-aafc35402300" containerID="255b9fa428fc6bf0e57985714327b234be69aa03eea9dc8a703ff3db0b7b1c66" exitCode=0
Nov 02 01:39:28.525856 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.525465    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-wks74"
Nov 02 01:39:28.525856 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.525491    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-wks74" event={"ID":"de494104-8e40-42d5-a2c1-aafc35402300","Type":"ContainerDied","Data":"255b9fa428fc6bf0e57985714327b234be69aa03eea9dc8a703ff3db0b7b1c66"}
Nov 02 01:39:28.525856 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.525541    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-wks74" event={"ID":"de494104-8e40-42d5-a2c1-aafc35402300","Type":"ContainerDied","Data":"c800b1bfd726a571c0272d55e87cbfbd994e95319340ce34cf483d237b05b519"}
Nov 02 01:39:28.525856 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.525563    5875 scope.go:122] "RemoveContainer" containerID="255b9fa428fc6bf0e57985714327b234be69aa03eea9dc8a703ff3db0b7b1c66"
Nov 02 01:39:28.535452 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.535433    5875 scope.go:122] "RemoveContainer" containerID="255b9fa428fc6bf0e57985714327b234be69aa03eea9dc8a703ff3db0b7b1c66"
Nov 02 01:39:28.535775 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:39:28.535752    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"255b9fa428fc6bf0e57985714327b234be69aa03eea9dc8a703ff3db0b7b1c66\": container with ID starting with 255b9fa428fc6bf0e57985714327b234be69aa03eea9dc8a703ff3db0b7b1c66 not found: ID does not exist" containerID="255b9fa428fc6bf0e57985714327b234be69aa03eea9dc8a703ff3db0b7b1c66"
Nov 02 01:39:28.535872 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.535787    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"255b9fa428fc6bf0e57985714327b234be69aa03eea9dc8a703ff3db0b7b1c66"} err="failed to get container status \"255b9fa428fc6bf0e57985714327b234be69aa03eea9dc8a703ff3db0b7b1c66\": rpc error: code = NotFound desc = could not find container \"255b9fa428fc6bf0e57985714327b234be69aa03eea9dc8a703ff3db0b7b1c66\": container with ID starting with 255b9fa428fc6bf0e57985714327b234be69aa03eea9dc8a703ff3db0b7b1c66 not found: ID does not exist"
Nov 02 01:39:28.677445 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.677413    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"]
Nov 02 01:39:28.677844 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.677831    5875 cpu_manager.go:414] "containerMap: removing container" podUID="de494104-8e40-42d5-a2c1-aafc35402300" containerName="manager"
Nov 02 01:39:28.677899 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.677847    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de494104-8e40-42d5-a2c1-aafc35402300" containerName="manager"
Nov 02 01:39:28.677934 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.677925    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="de494104-8e40-42d5-a2c1-aafc35402300" containerName="manager"
Nov 02 01:39:28.682344 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.682326    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"
Nov 02 01:39:28.695529 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.695509    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-wks74"]
Nov 02 01:39:28.727412 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.727375    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-wks74"]
Nov 02 01:39:28.734583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.734559    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 01:39:28.734583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.734577    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 01:39:28.734758 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.734560    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 01:39:28.734758 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.734557    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 01:39:28.734758 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.734620    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-f8tcq\""
Nov 02 01:39:28.785062 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.784972    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"]
Nov 02 01:39:28.801663 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.801635    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4ed38132-83a1-4cc5-9545-c1623a3b003f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8vnh\" (UID: \"4ed38132-83a1-4cc5-9545-c1623a3b003f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"
Nov 02 01:39:28.801835 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.801671    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ed38132-83a1-4cc5-9545-c1623a3b003f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8vnh\" (UID: \"4ed38132-83a1-4cc5-9545-c1623a3b003f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"
Nov 02 01:39:28.801835 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.801695    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed38132-83a1-4cc5-9545-c1623a3b003f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8vnh\" (UID: \"4ed38132-83a1-4cc5-9545-c1623a3b003f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"
Nov 02 01:39:28.801933 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.801914    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4ed38132-83a1-4cc5-9545-c1623a3b003f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8vnh\" (UID: \"4ed38132-83a1-4cc5-9545-c1623a3b003f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"
Nov 02 01:39:28.801966 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.801952    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4ed38132-83a1-4cc5-9545-c1623a3b003f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8vnh\" (UID: \"4ed38132-83a1-4cc5-9545-c1623a3b003f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"
Nov 02 01:39:28.802067 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.802047    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hgwc\" (UniqueName: \"kubernetes.io/projected/4ed38132-83a1-4cc5-9545-c1623a3b003f-kube-api-access-6hgwc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8vnh\" (UID: \"4ed38132-83a1-4cc5-9545-c1623a3b003f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"
Nov 02 01:39:28.802129 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.802085    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed38132-83a1-4cc5-9545-c1623a3b003f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8vnh\" (UID: \"4ed38132-83a1-4cc5-9545-c1623a3b003f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"
Nov 02 01:39:28.903174 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.903144    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed38132-83a1-4cc5-9545-c1623a3b003f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8vnh\" (UID: \"4ed38132-83a1-4cc5-9545-c1623a3b003f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"
Nov 02 01:39:28.903389 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.903250    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4ed38132-83a1-4cc5-9545-c1623a3b003f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8vnh\" (UID: \"4ed38132-83a1-4cc5-9545-c1623a3b003f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"
Nov 02 01:39:28.903389 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.903284    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4ed38132-83a1-4cc5-9545-c1623a3b003f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8vnh\" (UID: \"4ed38132-83a1-4cc5-9545-c1623a3b003f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"
Nov 02 01:39:28.903389 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.903320    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hgwc\" (UniqueName: \"kubernetes.io/projected/4ed38132-83a1-4cc5-9545-c1623a3b003f-kube-api-access-6hgwc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8vnh\" (UID: \"4ed38132-83a1-4cc5-9545-c1623a3b003f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"
Nov 02 01:39:28.903389 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.903346    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed38132-83a1-4cc5-9545-c1623a3b003f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8vnh\" (UID: \"4ed38132-83a1-4cc5-9545-c1623a3b003f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"
Nov 02 01:39:28.903389 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.903374    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4ed38132-83a1-4cc5-9545-c1623a3b003f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8vnh\" (UID: \"4ed38132-83a1-4cc5-9545-c1623a3b003f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"
Nov 02 01:39:28.903682 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.903404    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ed38132-83a1-4cc5-9545-c1623a3b003f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8vnh\" (UID: \"4ed38132-83a1-4cc5-9545-c1623a3b003f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"
Nov 02 01:39:28.904101 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.904070    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4ed38132-83a1-4cc5-9545-c1623a3b003f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8vnh\" (UID: \"4ed38132-83a1-4cc5-9545-c1623a3b003f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"
Nov 02 01:39:28.905889 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.905863    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed38132-83a1-4cc5-9545-c1623a3b003f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8vnh\" (UID: \"4ed38132-83a1-4cc5-9545-c1623a3b003f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"
Nov 02 01:39:28.906007 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.905990    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ed38132-83a1-4cc5-9545-c1623a3b003f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8vnh\" (UID: \"4ed38132-83a1-4cc5-9545-c1623a3b003f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"
Nov 02 01:39:28.906066 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.906033    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4ed38132-83a1-4cc5-9545-c1623a3b003f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8vnh\" (UID: \"4ed38132-83a1-4cc5-9545-c1623a3b003f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"
Nov 02 01:39:28.906120 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.906081    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4ed38132-83a1-4cc5-9545-c1623a3b003f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8vnh\" (UID: \"4ed38132-83a1-4cc5-9545-c1623a3b003f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"
Nov 02 01:39:28.964000 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.963933    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hgwc\" (UniqueName: \"kubernetes.io/projected/4ed38132-83a1-4cc5-9545-c1623a3b003f-kube-api-access-6hgwc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8vnh\" (UID: \"4ed38132-83a1-4cc5-9545-c1623a3b003f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"
Nov 02 01:39:28.964167 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.964129    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed38132-83a1-4cc5-9545-c1623a3b003f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t8vnh\" (UID: \"4ed38132-83a1-4cc5-9545-c1623a3b003f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"
Nov 02 01:39:28.996050 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:28.996013    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"
Nov 02 01:39:29.421291 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:29.421263    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"]
Nov 02 01:39:29.423796 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:39:29.423766    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed38132_83a1_4cc5_9545_c1623a3b003f.slice/crio-a28ef597d189ea7ff2687c3b39eeea7c145d92aa7f0eb1db19721a36b89c14ab WatchSource:0}: Error finding container a28ef597d189ea7ff2687c3b39eeea7c145d92aa7f0eb1db19721a36b89c14ab: Status 404 returned error can't find the container with id a28ef597d189ea7ff2687c3b39eeea7c145d92aa7f0eb1db19721a36b89c14ab
Nov 02 01:39:29.539036 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:29.538999    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh" event={"ID":"4ed38132-83a1-4cc5-9545-c1623a3b003f","Type":"ContainerStarted","Data":"a28ef597d189ea7ff2687c3b39eeea7c145d92aa7f0eb1db19721a36b89c14ab"}
Nov 02 01:39:30.692551 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:30.692517    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de494104-8e40-42d5-a2c1-aafc35402300" path="/var/lib/kubelet/pods/de494104-8e40-42d5-a2c1-aafc35402300/volumes"
Nov 02 01:39:33.378889 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:33.378857    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:39:33.379204 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:33.378931    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:39:33.560857 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:33.560818    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh" event={"ID":"4ed38132-83a1-4cc5-9545-c1623a3b003f","Type":"ContainerStarted","Data":"56b9e8ce14ed37b3a1d3b849d2fd3b656108169acda97bc40ef89e79d53d95a9"}
Nov 02 01:39:33.561064 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:33.560964    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"
Nov 02 01:39:33.778632 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:33.778573    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh" podStartSLOduration=1.8387432590000001 podStartE2EDuration="5.778559824s" totalImagesPullingTime="3.939816565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:39:33.778142491 +0000 UTC m=+421.219178990" watchObservedRunningTime="2026-11-02 01:39:33.778559824 +0000 UTC m=+421.219596320"
Nov 02 01:39:34.567130 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:34.567098    5875 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-t8vnh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:39:34.567575 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:34.567153    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh" podUID="4ed38132-83a1-4cc5-9545-c1623a3b003f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:39:37.566139 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:37.566102    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t8vnh"
Nov 02 01:39:41.317643 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:39:41.317601    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:39:58.501131 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:39:58.501082    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:40:08.755475 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:40:08.755434    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-z8lgq"]
Nov 02 01:40:08.755880 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:40:08.755681    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-z8lgq" podUID="f1f4adf7-97b0-4b57-a794-dfb041751d12" containerName="authorino" containerID="cri-o://19e311e4494a69339c66b721add91dc681859ae3b412cffb09aee354dcdb5ab3" gracePeriod=30
Nov 02 01:40:09.008823 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:40:09.008769    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-z8lgq"
Nov 02 01:40:09.080108 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:40:09.080077    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1f4adf7-97b0-4b57-a794-dfb041751d12-kube-api-access-cd5qg\" (UniqueName: \"kubernetes.io/projected/f1f4adf7-97b0-4b57-a794-dfb041751d12-kube-api-access-cd5qg\") pod \"f1f4adf7-97b0-4b57-a794-dfb041751d12\" (UID: \"f1f4adf7-97b0-4b57-a794-dfb041751d12\") "
Nov 02 01:40:09.082182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:40:09.082159    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1f4adf7-97b0-4b57-a794-dfb041751d12-kube-api-access-cd5qg" pod "f1f4adf7-97b0-4b57-a794-dfb041751d12" (UID: "f1f4adf7-97b0-4b57-a794-dfb041751d12"). InnerVolumeSpecName "kube-api-access-cd5qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:40:09.181544 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:40:09.181514    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd5qg\" (UniqueName: \"kubernetes.io/projected/f1f4adf7-97b0-4b57-a794-dfb041751d12-kube-api-access-cd5qg\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:40:09.732610 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:40:09.732576    5875 generic.go:356] "Generic (PLEG): container finished" podID="f1f4adf7-97b0-4b57-a794-dfb041751d12" containerID="19e311e4494a69339c66b721add91dc681859ae3b412cffb09aee354dcdb5ab3" exitCode=0
Nov 02 01:40:09.732789 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:40:09.732620    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-z8lgq"
Nov 02 01:40:09.732789 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:40:09.732628    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-z8lgq" event={"ID":"f1f4adf7-97b0-4b57-a794-dfb041751d12","Type":"ContainerDied","Data":"19e311e4494a69339c66b721add91dc681859ae3b412cffb09aee354dcdb5ab3"}
Nov 02 01:40:09.732789 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:40:09.732671    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-z8lgq" event={"ID":"f1f4adf7-97b0-4b57-a794-dfb041751d12","Type":"ContainerDied","Data":"fe51ae315285df6a758ea80695c6290fab54e9f73385ba9864d71f1e62185a6f"}
Nov 02 01:40:09.732789 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:40:09.732689    5875 scope.go:122] "RemoveContainer" containerID="19e311e4494a69339c66b721add91dc681859ae3b412cffb09aee354dcdb5ab3"
Nov 02 01:40:09.741363 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:40:09.741343    5875 scope.go:122] "RemoveContainer" containerID="19e311e4494a69339c66b721add91dc681859ae3b412cffb09aee354dcdb5ab3"
Nov 02 01:40:09.741626 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:40:09.741608    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19e311e4494a69339c66b721add91dc681859ae3b412cffb09aee354dcdb5ab3\": container with ID starting with 19e311e4494a69339c66b721add91dc681859ae3b412cffb09aee354dcdb5ab3 not found: ID does not exist" containerID="19e311e4494a69339c66b721add91dc681859ae3b412cffb09aee354dcdb5ab3"
Nov 02 01:40:09.741693 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:40:09.741639    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e311e4494a69339c66b721add91dc681859ae3b412cffb09aee354dcdb5ab3"} err="failed to get container status \"19e311e4494a69339c66b721add91dc681859ae3b412cffb09aee354dcdb5ab3\": rpc error: code = NotFound desc = could not find container \"19e311e4494a69339c66b721add91dc681859ae3b412cffb09aee354dcdb5ab3\": container with ID starting with 19e311e4494a69339c66b721add91dc681859ae3b412cffb09aee354dcdb5ab3 not found: ID does not exist"
Nov 02 01:40:09.894606 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:40:09.894575    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-z8lgq"]
Nov 02 01:40:09.943944 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:40:09.943918    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-z8lgq"]
Nov 02 01:40:10.692615 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:40:10.692578    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1f4adf7-97b0-4b57-a794-dfb041751d12" path="/var/lib/kubelet/pods/f1f4adf7-97b0-4b57-a794-dfb041751d12/volumes"
Nov 02 01:40:11.317207 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:40:11.317168    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:40:41.317141 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:40:41.317099    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:41:11.317840 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:41:11.317782    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:41:41.317996 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:41:41.317907    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:41:49.998787 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:41:49.998751    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:41:57.051870 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:41:57.051833    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:42:01.006412 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.006379    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"]
Nov 02 01:42:01.006783 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.006736    5875 cpu_manager.go:414] "containerMap: removing container" podUID="f1f4adf7-97b0-4b57-a794-dfb041751d12" containerName="authorino"
Nov 02 01:42:01.006783 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.006748    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1f4adf7-97b0-4b57-a794-dfb041751d12" containerName="authorino"
Nov 02 01:42:01.006856 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.006832    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1f4adf7-97b0-4b57-a794-dfb041751d12" containerName="authorino"
Nov 02 01:42:01.010025 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.010008    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"
Nov 02 01:42:01.064924 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.064901    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7vwh8\""
Nov 02 01:42:01.106514 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.106485    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 02 01:42:01.114394 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.114368    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 01:42:01.114394 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.114373    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 01:42:01.115769 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.115744    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"]
Nov 02 01:42:01.166668 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.166645    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d0bed6b-fa05-48dd-8f55-21d4903222c9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fxwst\" (UID: \"2d0bed6b-fa05-48dd-8f55-21d4903222c9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"
Nov 02 01:42:01.166791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.166679    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d0bed6b-fa05-48dd-8f55-21d4903222c9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fxwst\" (UID: \"2d0bed6b-fa05-48dd-8f55-21d4903222c9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"
Nov 02 01:42:01.166791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.166713    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j8j8\" (UniqueName: \"kubernetes.io/projected/2d0bed6b-fa05-48dd-8f55-21d4903222c9-kube-api-access-6j8j8\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fxwst\" (UID: \"2d0bed6b-fa05-48dd-8f55-21d4903222c9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"
Nov 02 01:42:01.166884 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.166806    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d0bed6b-fa05-48dd-8f55-21d4903222c9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fxwst\" (UID: \"2d0bed6b-fa05-48dd-8f55-21d4903222c9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"
Nov 02 01:42:01.166884 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.166869    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d0bed6b-fa05-48dd-8f55-21d4903222c9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fxwst\" (UID: \"2d0bed6b-fa05-48dd-8f55-21d4903222c9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"
Nov 02 01:42:01.166969 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.166922    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d0bed6b-fa05-48dd-8f55-21d4903222c9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fxwst\" (UID: \"2d0bed6b-fa05-48dd-8f55-21d4903222c9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"
Nov 02 01:42:01.268352 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.268274    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d0bed6b-fa05-48dd-8f55-21d4903222c9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fxwst\" (UID: \"2d0bed6b-fa05-48dd-8f55-21d4903222c9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"
Nov 02 01:42:01.268352 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.268349    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d0bed6b-fa05-48dd-8f55-21d4903222c9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fxwst\" (UID: \"2d0bed6b-fa05-48dd-8f55-21d4903222c9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"
Nov 02 01:42:01.268554 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.268378    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d0bed6b-fa05-48dd-8f55-21d4903222c9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fxwst\" (UID: \"2d0bed6b-fa05-48dd-8f55-21d4903222c9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"
Nov 02 01:42:01.268554 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.268417    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j8j8\" (UniqueName: \"kubernetes.io/projected/2d0bed6b-fa05-48dd-8f55-21d4903222c9-kube-api-access-6j8j8\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fxwst\" (UID: \"2d0bed6b-fa05-48dd-8f55-21d4903222c9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"
Nov 02 01:42:01.268554 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.268443    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d0bed6b-fa05-48dd-8f55-21d4903222c9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fxwst\" (UID: \"2d0bed6b-fa05-48dd-8f55-21d4903222c9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"
Nov 02 01:42:01.268554 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.268492    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d0bed6b-fa05-48dd-8f55-21d4903222c9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fxwst\" (UID: \"2d0bed6b-fa05-48dd-8f55-21d4903222c9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"
Nov 02 01:42:01.269116 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.268823    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d0bed6b-fa05-48dd-8f55-21d4903222c9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fxwst\" (UID: \"2d0bed6b-fa05-48dd-8f55-21d4903222c9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"
Nov 02 01:42:01.269116 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.268837    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d0bed6b-fa05-48dd-8f55-21d4903222c9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fxwst\" (UID: \"2d0bed6b-fa05-48dd-8f55-21d4903222c9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"
Nov 02 01:42:01.269116 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.268891    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d0bed6b-fa05-48dd-8f55-21d4903222c9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fxwst\" (UID: \"2d0bed6b-fa05-48dd-8f55-21d4903222c9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"
Nov 02 01:42:01.271046 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.271021    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d0bed6b-fa05-48dd-8f55-21d4903222c9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fxwst\" (UID: \"2d0bed6b-fa05-48dd-8f55-21d4903222c9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"
Nov 02 01:42:01.271366 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.271349    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d0bed6b-fa05-48dd-8f55-21d4903222c9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fxwst\" (UID: \"2d0bed6b-fa05-48dd-8f55-21d4903222c9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"
Nov 02 01:42:01.329087 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.329061    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j8j8\" (UniqueName: \"kubernetes.io/projected/2d0bed6b-fa05-48dd-8f55-21d4903222c9-kube-api-access-6j8j8\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fxwst\" (UID: \"2d0bed6b-fa05-48dd-8f55-21d4903222c9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"
Nov 02 01:42:01.622728 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.622642    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"
Nov 02 01:42:01.993383 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:01.993348    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"]
Nov 02 01:42:01.996431 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:42:01.996406    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0bed6b_fa05_48dd_8f55_21d4903222c9.slice/crio-64803a16bf37072f720f086304d8d776f5d950247e2e97be4b11578db2d0f9e0 WatchSource:0}: Error finding container 64803a16bf37072f720f086304d8d776f5d950247e2e97be4b11578db2d0f9e0: Status 404 returned error can't find the container with id 64803a16bf37072f720f086304d8d776f5d950247e2e97be4b11578db2d0f9e0
Nov 02 01:42:02.228100 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:02.228067    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst" event={"ID":"2d0bed6b-fa05-48dd-8f55-21d4903222c9","Type":"ContainerStarted","Data":"64803a16bf37072f720f086304d8d776f5d950247e2e97be4b11578db2d0f9e0"}
Nov 02 01:42:04.239262 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:04.239207    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst" event={"ID":"2d0bed6b-fa05-48dd-8f55-21d4903222c9","Type":"ContainerStarted","Data":"02af2daaa401e1dc18e5222e7398f023f3f21f1ccca22c04775de574db79087c"}
Nov 02 01:42:04.623083 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:04.623011    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"
Nov 02 01:42:05.078685 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:05.078597    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst" podStartSLOduration=3.573901511 podStartE2EDuration="5.078580183s" totalImagesPullingTime="1.504678672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:04.356911601 +0000 UTC m=+571.797948099" watchObservedRunningTime="2026-11-02 01:42:05.078580183 +0000 UTC m=+572.519616679"
Nov 02 01:42:05.080304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:05.080279    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:42:08.040725 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:08.040687    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"]
Nov 02 01:42:08.044490 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:08.044467    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"
Nov 02 01:42:08.103291 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:08.103262    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"]
Nov 02 01:42:08.133328 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:08.133302    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/368af2aa-46fd-4d17-b1c7-fcb9122e1632-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4d5r\" (UID: \"368af2aa-46fd-4d17-b1c7-fcb9122e1632\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"
Nov 02 01:42:08.133476 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:08.133350    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/368af2aa-46fd-4d17-b1c7-fcb9122e1632-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4d5r\" (UID: \"368af2aa-46fd-4d17-b1c7-fcb9122e1632\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"
Nov 02 01:42:08.133476 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:08.133443    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/368af2aa-46fd-4d17-b1c7-fcb9122e1632-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4d5r\" (UID: \"368af2aa-46fd-4d17-b1c7-fcb9122e1632\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"
Nov 02 01:42:08.133572 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:08.133499    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twqdw\" (UniqueName: \"kubernetes.io/projected/368af2aa-46fd-4d17-b1c7-fcb9122e1632-kube-api-access-twqdw\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4d5r\" (UID: \"368af2aa-46fd-4d17-b1c7-fcb9122e1632\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"
Nov 02 01:42:08.133632 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:08.133608    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/368af2aa-46fd-4d17-b1c7-fcb9122e1632-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4d5r\" (UID: \"368af2aa-46fd-4d17-b1c7-fcb9122e1632\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"
Nov 02 01:42:08.133676 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:08.133660    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/368af2aa-46fd-4d17-b1c7-fcb9122e1632-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4d5r\" (UID: \"368af2aa-46fd-4d17-b1c7-fcb9122e1632\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"
Nov 02 01:42:08.146077 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:08.146059    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 02 01:42:08.234405 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:08.234375    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/368af2aa-46fd-4d17-b1c7-fcb9122e1632-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4d5r\" (UID: \"368af2aa-46fd-4d17-b1c7-fcb9122e1632\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"
Nov 02 01:42:08.234571 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:08.234420    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/368af2aa-46fd-4d17-b1c7-fcb9122e1632-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4d5r\" (UID: \"368af2aa-46fd-4d17-b1c7-fcb9122e1632\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"
Nov 02 01:42:08.234571 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:08.234464    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twqdw\" (UniqueName: \"kubernetes.io/projected/368af2aa-46fd-4d17-b1c7-fcb9122e1632-kube-api-access-twqdw\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4d5r\" (UID: \"368af2aa-46fd-4d17-b1c7-fcb9122e1632\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"
Nov 02 01:42:08.234571 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:08.234501    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/368af2aa-46fd-4d17-b1c7-fcb9122e1632-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4d5r\" (UID: \"368af2aa-46fd-4d17-b1c7-fcb9122e1632\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"
Nov 02 01:42:08.234571 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:08.234540    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/368af2aa-46fd-4d17-b1c7-fcb9122e1632-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4d5r\" (UID: \"368af2aa-46fd-4d17-b1c7-fcb9122e1632\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"
Nov 02 01:42:08.234755 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:08.234614    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/368af2aa-46fd-4d17-b1c7-fcb9122e1632-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4d5r\" (UID: \"368af2aa-46fd-4d17-b1c7-fcb9122e1632\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"
Nov 02 01:42:08.234832 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:08.234810    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/368af2aa-46fd-4d17-b1c7-fcb9122e1632-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4d5r\" (UID: \"368af2aa-46fd-4d17-b1c7-fcb9122e1632\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"
Nov 02 01:42:08.234884 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:08.234864    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/368af2aa-46fd-4d17-b1c7-fcb9122e1632-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4d5r\" (UID: \"368af2aa-46fd-4d17-b1c7-fcb9122e1632\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"
Nov 02 01:42:08.234994 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:08.234978    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/368af2aa-46fd-4d17-b1c7-fcb9122e1632-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4d5r\" (UID: \"368af2aa-46fd-4d17-b1c7-fcb9122e1632\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"
Nov 02 01:42:08.236884 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:08.236847    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/368af2aa-46fd-4d17-b1c7-fcb9122e1632-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4d5r\" (UID: \"368af2aa-46fd-4d17-b1c7-fcb9122e1632\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"
Nov 02 01:42:08.237329 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:08.237305    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/368af2aa-46fd-4d17-b1c7-fcb9122e1632-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4d5r\" (UID: \"368af2aa-46fd-4d17-b1c7-fcb9122e1632\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"
Nov 02 01:42:08.295742 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:08.295676    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twqdw\" (UniqueName: \"kubernetes.io/projected/368af2aa-46fd-4d17-b1c7-fcb9122e1632-kube-api-access-twqdw\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4d5r\" (UID: \"368af2aa-46fd-4d17-b1c7-fcb9122e1632\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"
Nov 02 01:42:08.357530 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:08.357505    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"
Nov 02 01:42:08.731764 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:08.731735    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"]
Nov 02 01:42:08.734787 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:42:08.734762    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod368af2aa_46fd_4d17_b1c7_fcb9122e1632.slice/crio-f0f4c6a1e9ce86b83e3b23fcd03a879173bc388414140506658d00f26dffb162 WatchSource:0}: Error finding container f0f4c6a1e9ce86b83e3b23fcd03a879173bc388414140506658d00f26dffb162: Status 404 returned error can't find the container with id f0f4c6a1e9ce86b83e3b23fcd03a879173bc388414140506658d00f26dffb162
Nov 02 01:42:09.264825 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:09.264786    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r" event={"ID":"368af2aa-46fd-4d17-b1c7-fcb9122e1632","Type":"ContainerStarted","Data":"f0f4c6a1e9ce86b83e3b23fcd03a879173bc388414140506658d00f26dffb162"}
Nov 02 01:42:10.272693 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:10.272650    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r" event={"ID":"368af2aa-46fd-4d17-b1c7-fcb9122e1632","Type":"ContainerStarted","Data":"eeabd63086350d869dbcee16f74be803c4e857441ebb58dceb2f288545196914"}
Nov 02 01:42:10.358606 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:10.358576    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"
Nov 02 01:42:10.391135 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:10.391082    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r" podStartSLOduration=2.7977203680000002 podStartE2EDuration="3.391067412s" totalImagesPullingTime="593.347044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:10.390917966 +0000 UTC m=+577.831954460" watchObservedRunningTime="2026-11-02 01:42:10.391067412 +0000 UTC m=+577.832103908"
Nov 02 01:42:11.317809 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:42:11.317772    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:42:11.519589 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:11.519557    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"]
Nov 02 01:42:11.523153 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:11.523137    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"
Nov 02 01:42:11.560194 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:11.560165    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61e08deb-d43d-4418-8c8d-414c18fa0916-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn\" (UID: \"61e08deb-d43d-4418-8c8d-414c18fa0916\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"
Nov 02 01:42:11.560370 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:11.560266    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61e08deb-d43d-4418-8c8d-414c18fa0916-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn\" (UID: \"61e08deb-d43d-4418-8c8d-414c18fa0916\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"
Nov 02 01:42:11.560370 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:11.560321    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e08deb-d43d-4418-8c8d-414c18fa0916-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn\" (UID: \"61e08deb-d43d-4418-8c8d-414c18fa0916\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"
Nov 02 01:42:11.560370 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:11.560350    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-277df\" (UniqueName: \"kubernetes.io/projected/61e08deb-d43d-4418-8c8d-414c18fa0916-kube-api-access-277df\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn\" (UID: \"61e08deb-d43d-4418-8c8d-414c18fa0916\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"
Nov 02 01:42:11.560519 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:11.560403    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61e08deb-d43d-4418-8c8d-414c18fa0916-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn\" (UID: \"61e08deb-d43d-4418-8c8d-414c18fa0916\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"
Nov 02 01:42:11.560519 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:11.560432    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61e08deb-d43d-4418-8c8d-414c18fa0916-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn\" (UID: \"61e08deb-d43d-4418-8c8d-414c18fa0916\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"
Nov 02 01:42:11.623319 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:11.623241    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"
Nov 02 01:42:11.624425 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:11.624404    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 02 01:42:11.625803 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:11.625785    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"]
Nov 02 01:42:11.635620 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:11.635602    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"
Nov 02 01:42:11.661623 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:11.661597    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61e08deb-d43d-4418-8c8d-414c18fa0916-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn\" (UID: \"61e08deb-d43d-4418-8c8d-414c18fa0916\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"
Nov 02 01:42:11.661737 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:11.661705    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e08deb-d43d-4418-8c8d-414c18fa0916-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn\" (UID: \"61e08deb-d43d-4418-8c8d-414c18fa0916\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"
Nov 02 01:42:11.661808 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:11.661737    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-277df\" (UniqueName: \"kubernetes.io/projected/61e08deb-d43d-4418-8c8d-414c18fa0916-kube-api-access-277df\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn\" (UID: \"61e08deb-d43d-4418-8c8d-414c18fa0916\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"
Nov 02 01:42:11.661808 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:11.661780    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61e08deb-d43d-4418-8c8d-414c18fa0916-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn\" (UID: \"61e08deb-d43d-4418-8c8d-414c18fa0916\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"
Nov 02 01:42:11.661926 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:11.661891    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61e08deb-d43d-4418-8c8d-414c18fa0916-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn\" (UID: \"61e08deb-d43d-4418-8c8d-414c18fa0916\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"
Nov 02 01:42:11.661991 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:11.661977    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61e08deb-d43d-4418-8c8d-414c18fa0916-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn\" (UID: \"61e08deb-d43d-4418-8c8d-414c18fa0916\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"
Nov 02 01:42:11.662517 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:11.662147    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e08deb-d43d-4418-8c8d-414c18fa0916-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn\" (UID: \"61e08deb-d43d-4418-8c8d-414c18fa0916\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"
Nov 02 01:42:11.662517 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:11.662353    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61e08deb-d43d-4418-8c8d-414c18fa0916-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn\" (UID: \"61e08deb-d43d-4418-8c8d-414c18fa0916\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"
Nov 02 01:42:11.662658 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:11.662549    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61e08deb-d43d-4418-8c8d-414c18fa0916-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn\" (UID: \"61e08deb-d43d-4418-8c8d-414c18fa0916\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"
Nov 02 01:42:11.664263 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:11.664245    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61e08deb-d43d-4418-8c8d-414c18fa0916-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn\" (UID: \"61e08deb-d43d-4418-8c8d-414c18fa0916\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"
Nov 02 01:42:11.664659 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:11.664641    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61e08deb-d43d-4418-8c8d-414c18fa0916-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn\" (UID: \"61e08deb-d43d-4418-8c8d-414c18fa0916\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"
Nov 02 01:42:11.728258 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:11.728212    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-277df\" (UniqueName: \"kubernetes.io/projected/61e08deb-d43d-4418-8c8d-414c18fa0916-kube-api-access-277df\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn\" (UID: \"61e08deb-d43d-4418-8c8d-414c18fa0916\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"
Nov 02 01:42:11.836316 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:11.836286    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"
Nov 02 01:42:12.239578 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:12.239431    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"]
Nov 02 01:42:12.241716 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:42:12.241691    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e08deb_d43d_4418_8c8d_414c18fa0916.slice/crio-5a7fd24ee59c5b2763d385311f4db5aeabab915c1b4ea0e963110a32d325a731 WatchSource:0}: Error finding container 5a7fd24ee59c5b2763d385311f4db5aeabab915c1b4ea0e963110a32d325a731: Status 404 returned error can't find the container with id 5a7fd24ee59c5b2763d385311f4db5aeabab915c1b4ea0e963110a32d325a731
Nov 02 01:42:12.282290 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:12.282252    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn" event={"ID":"61e08deb-d43d-4418-8c8d-414c18fa0916","Type":"ContainerStarted","Data":"5a7fd24ee59c5b2763d385311f4db5aeabab915c1b4ea0e963110a32d325a731"}
Nov 02 01:42:12.293459 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:12.293432    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst"
Nov 02 01:42:13.289121 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:13.289083    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn" event={"ID":"61e08deb-d43d-4418-8c8d-414c18fa0916","Type":"ContainerStarted","Data":"56fe877d31704e0cc6afe62160e80fdfc5f42f16fdedb6f9c00aebbaf274561e"}
Nov 02 01:42:13.407021 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:13.406970    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn" podStartSLOduration=1.8344945190000002 podStartE2EDuration="2.406955066s" totalImagesPullingTime="572.460547ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:13.406873246 +0000 UTC m=+580.847909744" watchObservedRunningTime="2026-11-02 01:42:13.406955066 +0000 UTC m=+580.847991557"
Nov 02 01:42:13.837251 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:13.837204    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"
Nov 02 01:42:17.599148 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:17.599110    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:42:18.358552 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:18.358522    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"
Nov 02 01:42:18.370766 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:18.370745    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"
Nov 02 01:42:19.327973 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:19.327941    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r"
Nov 02 01:42:21.836572 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:21.836532    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"
Nov 02 01:42:21.849261 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:21.849210    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"
Nov 02 01:42:22.341050 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:22.341020    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn"
Nov 02 01:42:24.050020 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.049990    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"]
Nov 02 01:42:24.053988 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.053970    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.106267 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.106215    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-58w2j\""
Nov 02 01:42:24.157283 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.157255    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"]
Nov 02 01:42:24.187064 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.187028    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.187064 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.187067    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.187327 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.187135    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvq5b\" (UniqueName: \"kubernetes.io/projected/407d3a7d-bc65-4626-8c49-3cb5091c32c4-kube-api-access-dvq5b\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.187327 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.187191    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.187327 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.187289    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.187483 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.187354    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.187483 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.187398    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.187483 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.187426    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.187483 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.187465    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.288154 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.288119    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.288154 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.288164    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.288418 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.288184    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.288418 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.288260    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvq5b\" (UniqueName: \"kubernetes.io/projected/407d3a7d-bc65-4626-8c49-3cb5091c32c4-kube-api-access-dvq5b\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.288418 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.288287    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.288418 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.288368    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.288418 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.288394    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.288816 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.288423    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.288816 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.288447    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.288962 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.288878    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.289014 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.288972    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.289105 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.289075    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.289203 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.289184    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.289300 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.289215    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.290988 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.290967    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.291644 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.291623    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.349319 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.349246    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.349465 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.349343    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvq5b\" (UniqueName: \"kubernetes.io/projected/407d3a7d-bc65-4626-8c49-3cb5091c32c4-kube-api-access-dvq5b\") pod \"maas-default-gateway-openshift-default-778b7b5c47-rd589\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.367327 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.367295    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:24.797347 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:24.797324    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"]
Nov 02 01:42:24.799628 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:42:24.799601    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod407d3a7d_bc65_4626_8c49_3cb5091c32c4.slice/crio-58ef2d6418582dc2bb3f07da8bca30fb3e4ade835b10da77d006bda97a3a720e WatchSource:0}: Error finding container 58ef2d6418582dc2bb3f07da8bca30fb3e4ade835b10da77d006bda97a3a720e: Status 404 returned error can't find the container with id 58ef2d6418582dc2bb3f07da8bca30fb3e4ade835b10da77d006bda97a3a720e
Nov 02 01:42:25.347261 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:25.347205    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589" event={"ID":"407d3a7d-bc65-4626-8c49-3cb5091c32c4","Type":"ContainerStarted","Data":"58ef2d6418582dc2bb3f07da8bca30fb3e4ade835b10da77d006bda97a3a720e"}
Nov 02 01:42:29.244519 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:29.244485    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:42:29.244746 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:29.244553    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:42:29.244746 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:29.244580    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:42:29.369749 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:29.369713    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589" event={"ID":"407d3a7d-bc65-4626-8c49-3cb5091c32c4","Type":"ContainerStarted","Data":"9dbacd0fe02c38a82e961dc84897d35f964bafbe48d80c4df6ecc230190895a2"}
Nov 02 01:42:29.527782 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:29.527686    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589" podStartSLOduration=1.085798026 podStartE2EDuration="5.527670727s" totalImagesPullingTime="4.441872701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:29.527389597 +0000 UTC m=+596.968426095" watchObservedRunningTime="2026-11-02 01:42:29.527670727 +0000 UTC m=+596.968707225"
Nov 02 01:42:30.367590 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:30.367549    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:30.368825 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:30.368799    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-rd589 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 01:42:30.368941 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:30.368849    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589" podUID="407d3a7d-bc65-4626-8c49-3cb5091c32c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 01:42:31.368580 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:31.368544    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-rd589 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 01:42:31.368969 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:31.368610    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589" podUID="407d3a7d-bc65-4626-8c49-3cb5091c32c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 01:42:32.368086 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:32.368049    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-rd589 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 01:42:32.368281 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:32.368116    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589" podUID="407d3a7d-bc65-4626-8c49-3cb5091c32c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 01:42:33.368600 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:33.368566    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-rd589 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 01:42:33.369066 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:33.368624    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589" podUID="407d3a7d-bc65-4626-8c49-3cb5091c32c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 01:42:34.368422 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:34.368389    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-rd589 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 01:42:34.368609 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:34.368449    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589" podUID="407d3a7d-bc65-4626-8c49-3cb5091c32c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 01:42:35.367875 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:35.367837    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-rd589 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 01:42:35.368296 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:35.367900    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589" podUID="407d3a7d-bc65-4626-8c49-3cb5091c32c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 01:42:36.368146 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:36.368109    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-rd589 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 01:42:36.368580 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:36.368164    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589" podUID="407d3a7d-bc65-4626-8c49-3cb5091c32c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 01:42:37.437925 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:37.437895    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:37.438995 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:37.438974    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:37.439421 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:37.439401    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:42:39.179179 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.179145    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"]
Nov 02 01:42:39.185697 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.185674    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.286691 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.286663    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"]
Nov 02 01:42:39.352986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.352952    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.352986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.352982    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.353281 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.353008    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.353281 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.353090    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.353281 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.353275    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.353421 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.353323    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b789m\" (UniqueName: \"kubernetes.io/projected/bafe4c70-f1ef-423f-a94c-552f5545ad6d-kube-api-access-b789m\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.353421 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.353378    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.353485 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.353449    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.353485 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.353471    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.454948 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.454916    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.455133 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.454963    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b789m\" (UniqueName: \"kubernetes.io/projected/bafe4c70-f1ef-423f-a94c-552f5545ad6d-kube-api-access-b789m\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.455133 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.454980    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.455133 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.454999    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.455133 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.455117    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.455404 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.455179    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.455404 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.455293    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.455404 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.455341    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.455404 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.455394    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.455404 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.455398    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.455659 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.455457    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.455723 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.455671    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.455779 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.455734    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.456262 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.456208    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.457491 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.457467    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.457621 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.457604    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.516413 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.516373    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.516568 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.516497    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b789m\" (UniqueName: \"kubernetes.io/projected/bafe4c70-f1ef-423f-a94c-552f5545ad6d-kube-api-access-b789m\") pod \"maas-default-gateway-openshift-default-778b7b5c47-gcng9\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:39.799257 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:39.799140    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:40.226745 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:40.226697    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"]
Nov 02 01:42:40.229428 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:42:40.229402    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbafe4c70_f1ef_423f_a94c_552f5545ad6d.slice/crio-e48cf041dece4a656fb2f91104130fc588eca2b4eea95cd8792444edd03b8860 WatchSource:0}: Error finding container e48cf041dece4a656fb2f91104130fc588eca2b4eea95cd8792444edd03b8860: Status 404 returned error can't find the container with id e48cf041dece4a656fb2f91104130fc588eca2b4eea95cd8792444edd03b8860
Nov 02 01:42:40.231637 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:40.231592    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:42:40.231742 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:40.231678    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:42:40.231742 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:40.231723    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:42:40.426279 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:40.426245    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9" event={"ID":"bafe4c70-f1ef-423f-a94c-552f5545ad6d","Type":"ContainerStarted","Data":"869871651f445395b7e5f1f979bee2a3b2a358a38df394218c5b1d028cd35e41"}
Nov 02 01:42:40.426473 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:40.426300    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9" event={"ID":"bafe4c70-f1ef-423f-a94c-552f5545ad6d","Type":"ContainerStarted","Data":"e48cf041dece4a656fb2f91104130fc588eca2b4eea95cd8792444edd03b8860"}
Nov 02 01:42:40.545805 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:40.545682    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9" podStartSLOduration=1.5456624639999998 podStartE2EDuration="1.545662464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:40.545512911 +0000 UTC m=+607.986549410" watchObservedRunningTime="2026-11-02 01:42:40.545662464 +0000 UTC m=+607.986698961"
Nov 02 01:42:40.665282 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:40.665249    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-dlvzn"]
Nov 02 01:42:40.669301 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:40.669279    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697b76b867-dlvzn"
Nov 02 01:42:40.728066 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:40.727937    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 02 01:42:40.731330 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:40.731238    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-dlvzn"]
Nov 02 01:42:40.731508 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:40.731344    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 02 01:42:40.731508 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:40.731405    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-xljhj\""
Nov 02 01:42:40.769216 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:40.769174    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lr2s\" (UniqueName: \"kubernetes.io/projected/9d39e34a-64c8-4960-9381-16e8dd8403e9-kube-api-access-4lr2s\") pod \"ai-gateway-controller-697b76b867-dlvzn\" (UID: \"9d39e34a-64c8-4960-9381-16e8dd8403e9\") " pod="opendatahub/ai-gateway-controller-697b76b867-dlvzn"
Nov 02 01:42:40.769434 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:40.769263    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d39e34a-64c8-4960-9381-16e8dd8403e9-metrics-certs\") pod \"ai-gateway-controller-697b76b867-dlvzn\" (UID: \"9d39e34a-64c8-4960-9381-16e8dd8403e9\") " pod="opendatahub/ai-gateway-controller-697b76b867-dlvzn"
Nov 02 01:42:40.822300 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:40.821512    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:40.870895 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:40.870863    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lr2s\" (UniqueName: \"kubernetes.io/projected/9d39e34a-64c8-4960-9381-16e8dd8403e9-kube-api-access-4lr2s\") pod \"ai-gateway-controller-697b76b867-dlvzn\" (UID: \"9d39e34a-64c8-4960-9381-16e8dd8403e9\") " pod="opendatahub/ai-gateway-controller-697b76b867-dlvzn"
Nov 02 01:42:40.871087 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:40.870905    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d39e34a-64c8-4960-9381-16e8dd8403e9-metrics-certs\") pod \"ai-gateway-controller-697b76b867-dlvzn\" (UID: \"9d39e34a-64c8-4960-9381-16e8dd8403e9\") " pod="opendatahub/ai-gateway-controller-697b76b867-dlvzn"
Nov 02 01:42:40.873569 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:40.873541    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d39e34a-64c8-4960-9381-16e8dd8403e9-metrics-certs\") pod \"ai-gateway-controller-697b76b867-dlvzn\" (UID: \"9d39e34a-64c8-4960-9381-16e8dd8403e9\") " pod="opendatahub/ai-gateway-controller-697b76b867-dlvzn"
Nov 02 01:42:40.932327 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:40.932292    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lr2s\" (UniqueName: \"kubernetes.io/projected/9d39e34a-64c8-4960-9381-16e8dd8403e9-kube-api-access-4lr2s\") pod \"ai-gateway-controller-697b76b867-dlvzn\" (UID: \"9d39e34a-64c8-4960-9381-16e8dd8403e9\") " pod="opendatahub/ai-gateway-controller-697b76b867-dlvzn"
Nov 02 01:42:40.987183 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:40.987151    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697b76b867-dlvzn"
Nov 02 01:42:41.344539 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:42:41.344503    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:42:41.386566 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:41.386456    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-dlvzn"]
Nov 02 01:42:41.389294 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:42:41.389263    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d39e34a_64c8_4960_9381_16e8dd8403e9.slice/crio-065f89d8428a8e7797d1473975eb40b59839f6ce75aec630ccef61e732bbe4fc WatchSource:0}: Error finding container 065f89d8428a8e7797d1473975eb40b59839f6ce75aec630ccef61e732bbe4fc: Status 404 returned error can't find the container with id 065f89d8428a8e7797d1473975eb40b59839f6ce75aec630ccef61e732bbe4fc
Nov 02 01:42:41.390650 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:41.390630    5875 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:42:41.432622 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:41.432589    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697b76b867-dlvzn" event={"ID":"9d39e34a-64c8-4960-9381-16e8dd8403e9","Type":"ContainerStarted","Data":"065f89d8428a8e7797d1473975eb40b59839f6ce75aec630ccef61e732bbe4fc"}
Nov 02 01:42:41.818692 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:41.818346    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-gcng9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 02 01:42:41.818692 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:41.818409    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9" podUID="bafe4c70-f1ef-423f-a94c-552f5545ad6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 02 01:42:42.813248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:42.807646    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-gcng9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 02 01:42:42.813248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:42.807715    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9" podUID="bafe4c70-f1ef-423f-a94c-552f5545ad6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 02 01:42:43.808248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:43.806574    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-gcng9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 02 01:42:43.808248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:43.806638    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9" podUID="bafe4c70-f1ef-423f-a94c-552f5545ad6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 02 01:42:44.812651 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:44.812613    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-gcng9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 02 01:42:44.813268 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:44.813205    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9" podUID="bafe4c70-f1ef-423f-a94c-552f5545ad6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 02 01:42:45.811247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:45.805592    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-gcng9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 02 01:42:45.811247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:45.805658    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9" podUID="bafe4c70-f1ef-423f-a94c-552f5545ad6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 02 01:42:46.799576 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:46.799539    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-gcng9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 02 01:42:46.800073 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:46.799595    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9" podUID="bafe4c70-f1ef-423f-a94c-552f5545ad6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 02 01:42:47.800423 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:47.800338    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-gcng9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 02 01:42:47.800423 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:47.800415    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9" podUID="bafe4c70-f1ef-423f-a94c-552f5545ad6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 02 01:42:48.471264 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:48.471197    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697b76b867-dlvzn" event={"ID":"9d39e34a-64c8-4960-9381-16e8dd8403e9","Type":"ContainerStarted","Data":"99b8c5e8c3f81e22e041f2f3c3794a134aea9ba3827a0e344efa4fb2d21a325a"}
Nov 02 01:42:48.471588 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:48.471568    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-697b76b867-dlvzn"
Nov 02 01:42:48.588644 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:48.588582    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-697b76b867-dlvzn" podStartSLOduration=2.438448009 podStartE2EDuration="8.588568411s" totalImagesPullingTime="6.150120402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:48.587299662 +0000 UTC m=+616.028336162" watchObservedRunningTime="2026-11-02 01:42:48.588568411 +0000 UTC m=+616.029604907"
Nov 02 01:42:48.803600 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:48.803508    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:48.804100 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:48.803778    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:48.804646 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:48.804626    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:42:59.478460 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:59.478426    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-697b76b867-dlvzn"
Nov 02 01:42:59.854874 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:59.854801    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7bbc5d984c-7w9xv"]
Nov 02 01:42:59.861083 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:59.861066    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7bbc5d984c-7w9xv"
Nov 02 01:42:59.917113 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:59.917082    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7bbc5d984c-7w9xv"]
Nov 02 01:42:59.983248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:59.983185    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9810c039-61bd-483f-9b96-857fbcb2f633-metrics-certs\") pod \"ai-gateway-controller-7bbc5d984c-7w9xv\" (UID: \"9810c039-61bd-483f-9b96-857fbcb2f633\") " pod="opendatahub/ai-gateway-controller-7bbc5d984c-7w9xv"
Nov 02 01:42:59.983452 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:42:59.983265    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r9gf\" (UniqueName: \"kubernetes.io/projected/9810c039-61bd-483f-9b96-857fbcb2f633-kube-api-access-9r9gf\") pod \"ai-gateway-controller-7bbc5d984c-7w9xv\" (UID: \"9810c039-61bd-483f-9b96-857fbcb2f633\") " pod="opendatahub/ai-gateway-controller-7bbc5d984c-7w9xv"
Nov 02 01:43:00.084209 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:00.084179    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9810c039-61bd-483f-9b96-857fbcb2f633-metrics-certs\") pod \"ai-gateway-controller-7bbc5d984c-7w9xv\" (UID: \"9810c039-61bd-483f-9b96-857fbcb2f633\") " pod="opendatahub/ai-gateway-controller-7bbc5d984c-7w9xv"
Nov 02 01:43:00.084209 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:00.084252    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r9gf\" (UniqueName: \"kubernetes.io/projected/9810c039-61bd-483f-9b96-857fbcb2f633-kube-api-access-9r9gf\") pod \"ai-gateway-controller-7bbc5d984c-7w9xv\" (UID: \"9810c039-61bd-483f-9b96-857fbcb2f633\") " pod="opendatahub/ai-gateway-controller-7bbc5d984c-7w9xv"
Nov 02 01:43:00.086745 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:00.086717    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9810c039-61bd-483f-9b96-857fbcb2f633-metrics-certs\") pod \"ai-gateway-controller-7bbc5d984c-7w9xv\" (UID: \"9810c039-61bd-483f-9b96-857fbcb2f633\") " pod="opendatahub/ai-gateway-controller-7bbc5d984c-7w9xv"
Nov 02 01:43:00.145665 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:00.145595    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r9gf\" (UniqueName: \"kubernetes.io/projected/9810c039-61bd-483f-9b96-857fbcb2f633-kube-api-access-9r9gf\") pod \"ai-gateway-controller-7bbc5d984c-7w9xv\" (UID: \"9810c039-61bd-483f-9b96-857fbcb2f633\") " pod="opendatahub/ai-gateway-controller-7bbc5d984c-7w9xv"
Nov 02 01:43:00.174649 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:00.174617    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7bbc5d984c-7w9xv"
Nov 02 01:43:00.553182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:00.553152    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7bbc5d984c-7w9xv"]
Nov 02 01:43:00.557182 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:43:00.557149    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9810c039_61bd_483f_9b96_857fbcb2f633.slice/crio-5af62e664401e6cbfdc9e6a50e247ac8674a1b7890e9bb7c66b163f42fd85611 WatchSource:0}: Error finding container 5af62e664401e6cbfdc9e6a50e247ac8674a1b7890e9bb7c66b163f42fd85611: Status 404 returned error can't find the container with id 5af62e664401e6cbfdc9e6a50e247ac8674a1b7890e9bb7c66b163f42fd85611
Nov 02 01:43:01.538096 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:01.538062    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7bbc5d984c-7w9xv" event={"ID":"9810c039-61bd-483f-9b96-857fbcb2f633","Type":"ContainerStarted","Data":"764be5b9000c85b6015329b72311aadc721c75607b56398cb7fa04ef4bee47b4"}
Nov 02 01:43:01.538292 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:01.538122    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7bbc5d984c-7w9xv" event={"ID":"9810c039-61bd-483f-9b96-857fbcb2f633","Type":"ContainerStarted","Data":"5af62e664401e6cbfdc9e6a50e247ac8674a1b7890e9bb7c66b163f42fd85611"}
Nov 02 01:43:01.538292 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:01.538150    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7bbc5d984c-7w9xv"
Nov 02 01:43:01.657816 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:01.657716    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7bbc5d984c-7w9xv" podStartSLOduration=1.842366292 podStartE2EDuration="2.657701865s" totalImagesPullingTime="815.335573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:01.656096529 +0000 UTC m=+629.097133027" watchObservedRunningTime="2026-11-02 01:43:01.657701865 +0000 UTC m=+629.098738359"
Nov 02 01:43:11.318066 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:43:11.317985    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:43:12.545663 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:12.545627    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7bbc5d984c-7w9xv"
Nov 02 01:43:12.716018 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:12.715989    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-dlvzn"]
Nov 02 01:43:12.716276 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:12.716248    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-697b76b867-dlvzn" podUID="9d39e34a-64c8-4960-9381-16e8dd8403e9" containerName="manager" containerID="cri-o://99b8c5e8c3f81e22e041f2f3c3794a134aea9ba3827a0e344efa4fb2d21a325a" gracePeriod=30
Nov 02 01:43:12.973016 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:12.972993    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697b76b867-dlvzn"
Nov 02 01:43:13.112123 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:13.112049    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d39e34a-64c8-4960-9381-16e8dd8403e9-kube-api-access-4lr2s\" (UniqueName: \"kubernetes.io/projected/9d39e34a-64c8-4960-9381-16e8dd8403e9-kube-api-access-4lr2s\") pod \"9d39e34a-64c8-4960-9381-16e8dd8403e9\" (UID: \"9d39e34a-64c8-4960-9381-16e8dd8403e9\") "
Nov 02 01:43:13.112269 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:13.112189    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d39e34a-64c8-4960-9381-16e8dd8403e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d39e34a-64c8-4960-9381-16e8dd8403e9-metrics-certs\") pod \"9d39e34a-64c8-4960-9381-16e8dd8403e9\" (UID: \"9d39e34a-64c8-4960-9381-16e8dd8403e9\") "
Nov 02 01:43:13.114435 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:13.114400    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d39e34a-64c8-4960-9381-16e8dd8403e9-metrics-certs" pod "9d39e34a-64c8-4960-9381-16e8dd8403e9" (UID: "9d39e34a-64c8-4960-9381-16e8dd8403e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:43:13.114435 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:13.114420    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d39e34a-64c8-4960-9381-16e8dd8403e9-kube-api-access-4lr2s" pod "9d39e34a-64c8-4960-9381-16e8dd8403e9" (UID: "9d39e34a-64c8-4960-9381-16e8dd8403e9"). InnerVolumeSpecName "kube-api-access-4lr2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:43:13.213751 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:13.213713    5875 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d39e34a-64c8-4960-9381-16e8dd8403e9-metrics-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:13.213751 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:13.213750    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lr2s\" (UniqueName: \"kubernetes.io/projected/9d39e34a-64c8-4960-9381-16e8dd8403e9-kube-api-access-4lr2s\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:43:13.603566 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:13.603535    5875 generic.go:356] "Generic (PLEG): container finished" podID="9d39e34a-64c8-4960-9381-16e8dd8403e9" containerID="99b8c5e8c3f81e22e041f2f3c3794a134aea9ba3827a0e344efa4fb2d21a325a" exitCode=0
Nov 02 01:43:13.603983 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:13.603584    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-697b76b867-dlvzn"
Nov 02 01:43:13.603983 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:13.603627    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697b76b867-dlvzn" event={"ID":"9d39e34a-64c8-4960-9381-16e8dd8403e9","Type":"ContainerDied","Data":"99b8c5e8c3f81e22e041f2f3c3794a134aea9ba3827a0e344efa4fb2d21a325a"}
Nov 02 01:43:13.603983 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:13.603685    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-697b76b867-dlvzn" event={"ID":"9d39e34a-64c8-4960-9381-16e8dd8403e9","Type":"ContainerDied","Data":"065f89d8428a8e7797d1473975eb40b59839f6ce75aec630ccef61e732bbe4fc"}
Nov 02 01:43:13.603983 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:13.603717    5875 scope.go:122] "RemoveContainer" containerID="99b8c5e8c3f81e22e041f2f3c3794a134aea9ba3827a0e344efa4fb2d21a325a"
Nov 02 01:43:13.612614 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:13.612598    5875 scope.go:122] "RemoveContainer" containerID="99b8c5e8c3f81e22e041f2f3c3794a134aea9ba3827a0e344efa4fb2d21a325a"
Nov 02 01:43:13.612896 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:43:13.612877    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99b8c5e8c3f81e22e041f2f3c3794a134aea9ba3827a0e344efa4fb2d21a325a\": container with ID starting with 99b8c5e8c3f81e22e041f2f3c3794a134aea9ba3827a0e344efa4fb2d21a325a not found: ID does not exist" containerID="99b8c5e8c3f81e22e041f2f3c3794a134aea9ba3827a0e344efa4fb2d21a325a"
Nov 02 01:43:13.612972 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:13.612910    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b8c5e8c3f81e22e041f2f3c3794a134aea9ba3827a0e344efa4fb2d21a325a"} err="failed to get container status \"99b8c5e8c3f81e22e041f2f3c3794a134aea9ba3827a0e344efa4fb2d21a325a\": rpc error: code = NotFound desc = could not find container \"99b8c5e8c3f81e22e041f2f3c3794a134aea9ba3827a0e344efa4fb2d21a325a\": container with ID starting with 99b8c5e8c3f81e22e041f2f3c3794a134aea9ba3827a0e344efa4fb2d21a325a not found: ID does not exist"
Nov 02 01:43:13.768465 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:13.768432    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-dlvzn"]
Nov 02 01:43:13.817998 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:13.817960    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-697b76b867-dlvzn"]
Nov 02 01:43:14.692770 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:14.692735    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d39e34a-64c8-4960-9381-16e8dd8403e9" path="/var/lib/kubelet/pods/9d39e34a-64c8-4960-9381-16e8dd8403e9/volumes"
Nov 02 01:43:41.317410 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:43:41.317372    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:43:49.006104 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.006067    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"]
Nov 02 01:43:49.006847 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.006825    5875 cpu_manager.go:414] "containerMap: removing container" podUID="9d39e34a-64c8-4960-9381-16e8dd8403e9" containerName="manager"
Nov 02 01:43:49.006890 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.006854    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d39e34a-64c8-4960-9381-16e8dd8403e9" containerName="manager"
Nov 02 01:43:49.007031 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.007015    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d39e34a-64c8-4960-9381-16e8dd8403e9" containerName="manager"
Nov 02 01:43:49.010666 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.010644    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.055792 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.055757    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-workload-certs\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.055965 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.055801    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f7e0b6-dc17-4891-9817-242be8465c54-istio-token\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.055965 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.055907    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f7e0b6-dc17-4891-9817-242be8465c54-istiod-ca-cert\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.056051 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.055991    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjczf\" (UniqueName: \"kubernetes.io/projected/05f7e0b6-dc17-4891-9817-242be8465c54-kube-api-access-kjczf\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.056051 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.056036    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f7e0b6-dc17-4891-9817-242be8465c54-istio-podinfo\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.056146 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.056134    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-istio-data\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.056200 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.056185    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-credential-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.056321 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.056302    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-istio-envoy\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.056375 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.056345    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-workload-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.069733 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.069701    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-0ebdb3-openshift-default-dockercfg-hkbq9\""
Nov 02 01:43:49.120759 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.120727    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"]
Nov 02 01:43:49.157634 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.157602    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-istio-data\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.157828 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.157652    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-credential-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.157828 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.157689    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-istio-envoy\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.157828 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.157721    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-workload-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.157828 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.157777    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-workload-certs\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.158027 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.157886    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f7e0b6-dc17-4891-9817-242be8465c54-istio-token\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.158027 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.157940    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f7e0b6-dc17-4891-9817-242be8465c54-istiod-ca-cert\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.158027 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.157996    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjczf\" (UniqueName: \"kubernetes.io/projected/05f7e0b6-dc17-4891-9817-242be8465c54-kube-api-access-kjczf\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.158166 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.158052    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f7e0b6-dc17-4891-9817-242be8465c54-istio-podinfo\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.158166 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.158084    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-istio-data\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.158166 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.158109    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-workload-certs\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.158166 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.158146    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-credential-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.158388 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.158243    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-workload-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.158676 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.158657    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f7e0b6-dc17-4891-9817-242be8465c54-istiod-ca-cert\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.160276 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.160255    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-istio-envoy\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.160492 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.160467    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f7e0b6-dc17-4891-9817-242be8465c54-istio-podinfo\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.224307 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.224271    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f7e0b6-dc17-4891-9817-242be8465c54-istio-token\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.224453 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.224368    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjczf\" (UniqueName: \"kubernetes.io/projected/05f7e0b6-dc17-4891-9817-242be8465c54-kube-api-access-kjczf\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.325381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.325290    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:49.713481 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.713374    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"]
Nov 02 01:43:49.720645 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.720610    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:49.780980 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.780915    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-061fab-openshift-default-dockercfg-k8pns\""
Nov 02 01:43:49.781343 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.781002    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"]
Nov 02 01:43:49.782910 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.782886    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"]
Nov 02 01:43:49.784556 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:43:49.784509    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f7e0b6_dc17_4891_9817_242be8465c54.slice/crio-8b4eca52e36194a9c62fe1939f70a6e20ab14aadd4cbc499ad28d3a0e493bc60 WatchSource:0}: Error finding container 8b4eca52e36194a9c62fe1939f70a6e20ab14aadd4cbc499ad28d3a0e493bc60: Status 404 returned error can't find the container with id 8b4eca52e36194a9c62fe1939f70a6e20ab14aadd4cbc499ad28d3a0e493bc60
Nov 02 01:43:49.787463 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.787431    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:43:49.787564 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.787495    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:43:49.787564 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.787523    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:43:49.866344 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.866310    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-workload-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:49.866451 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.866364    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/166343e8-524c-4668-9094-a503991be4f9-istio-podinfo\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:49.866513 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.866495    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-credential-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:49.866566 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.866551    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-istio-envoy\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:49.866655 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.866635    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-workload-certs\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:49.866689 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.866673    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jw7k\" (UniqueName: \"kubernetes.io/projected/166343e8-524c-4668-9094-a503991be4f9-kube-api-access-4jw7k\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:49.866721 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.866695    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-istio-data\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:49.866784 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.866771    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/166343e8-524c-4668-9094-a503991be4f9-istiod-ca-cert\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:49.866819 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.866796    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/166343e8-524c-4668-9094-a503991be4f9-istio-token\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:49.967486 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.967391    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-istio-envoy\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:49.967486 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.967469    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-workload-certs\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:49.967715 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.967495    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jw7k\" (UniqueName: \"kubernetes.io/projected/166343e8-524c-4668-9094-a503991be4f9-kube-api-access-4jw7k\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:49.967715 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.967517    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-istio-data\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:49.967715 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.967543    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/166343e8-524c-4668-9094-a503991be4f9-istiod-ca-cert\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:49.967715 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.967676    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/166343e8-524c-4668-9094-a503991be4f9-istio-token\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:49.967935 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.967724    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-workload-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:49.967935 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.967757    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/166343e8-524c-4668-9094-a503991be4f9-istio-podinfo\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:49.967935 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.967858    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-credential-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:49.967935 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.967906    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-workload-certs\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:49.967935 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.967926    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-istio-data\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:49.968371 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.968136    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-workload-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:49.968371 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.968319    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-credential-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:49.968371 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.968321    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/166343e8-524c-4668-9094-a503991be4f9-istiod-ca-cert\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:49.969819 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.969798    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-istio-envoy\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:49.970256 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:49.970215    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/166343e8-524c-4668-9094-a503991be4f9-istio-podinfo\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:50.029426 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.029391    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/166343e8-524c-4668-9094-a503991be4f9-istio-token\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:50.030010 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.029988    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jw7k\" (UniqueName: \"kubernetes.io/projected/166343e8-524c-4668-9094-a503991be4f9-kube-api-access-4jw7k\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:50.036200 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.036174    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:50.467689 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.467653    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"]
Nov 02 01:43:50.471452 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:43:50.471424    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod166343e8_524c_4668_9094_a503991be4f9.slice/crio-b2465df631c50378199cfb56d7a4ad8de3bb3dd3b35f37dd702fa6df8dfab655 WatchSource:0}: Error finding container b2465df631c50378199cfb56d7a4ad8de3bb3dd3b35f37dd702fa6df8dfab655: Status 404 returned error can't find the container with id b2465df631c50378199cfb56d7a4ad8de3bb3dd3b35f37dd702fa6df8dfab655
Nov 02 01:43:50.473699 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.473665    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:43:50.473775 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.473735    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:43:50.473775 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.473767    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:43:50.624686 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.624649    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"]
Nov 02 01:43:50.629131 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.629105    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.675125 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.675094    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-workload-socket\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.675325 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.675129    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-workload-certs\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.675325 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.675156    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th8gz\" (UniqueName: \"kubernetes.io/projected/c1cf6bdd-85db-4669-a694-ac7ab7445792-kube-api-access-th8gz\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.675447 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.675341    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1cf6bdd-85db-4669-a694-ac7ab7445792-istiod-ca-cert\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.675447 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.675411    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-podinfo\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.675447 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.675440    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-data\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.675577 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.675545    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-token\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.675651 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.675631    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-credential-socket\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.675717 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.675698    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-envoy\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.681859 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.681837    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-dbe849-openshift-default-dockercfg-7b7d6\""
Nov 02 01:43:50.696051 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.696026    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"]
Nov 02 01:43:50.776451 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.776350    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th8gz\" (UniqueName: \"kubernetes.io/projected/c1cf6bdd-85db-4669-a694-ac7ab7445792-kube-api-access-th8gz\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.776451 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.776413    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1cf6bdd-85db-4669-a694-ac7ab7445792-istiod-ca-cert\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.777730 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.776505    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-podinfo\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.777730 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.776541    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-data\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.777730 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.777200    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-token\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.777730 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.777210    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1cf6bdd-85db-4669-a694-ac7ab7445792-istiod-ca-cert\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.777730 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.777301    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-credential-socket\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.777730 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.777378    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-envoy\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.777730 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.777446    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-data\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.777730 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.777584    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-workload-socket\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.777730 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.777631    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-workload-certs\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.778292 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.777831    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-workload-socket\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.778292 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.778057    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-credential-socket\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.778522 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.778206    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-workload-certs\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.780400 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.780352    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-envoy\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.780972 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.780951    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-podinfo\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.787097 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.787071    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh" event={"ID":"05f7e0b6-dc17-4891-9817-242be8465c54","Type":"ContainerStarted","Data":"295e19acbb53caf69ec7f0f20cf688103e7b889e62d320dc8c7e525a17c1f33d"}
Nov 02 01:43:50.787265 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.787131    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh" event={"ID":"05f7e0b6-dc17-4891-9817-242be8465c54","Type":"ContainerStarted","Data":"8b4eca52e36194a9c62fe1939f70a6e20ab14aadd4cbc499ad28d3a0e493bc60"}
Nov 02 01:43:50.789107 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.789084    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw" event={"ID":"166343e8-524c-4668-9094-a503991be4f9","Type":"ContainerStarted","Data":"8f3f00f580c3ccd7ee044a5da972c67d9186da7c6a00b8cb8c07d52dc9b8b51d"}
Nov 02 01:43:50.789212 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.789129    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw" event={"ID":"166343e8-524c-4668-9094-a503991be4f9","Type":"ContainerStarted","Data":"b2465df631c50378199cfb56d7a4ad8de3bb3dd3b35f37dd702fa6df8dfab655"}
Nov 02 01:43:50.844113 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.844081    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th8gz\" (UniqueName: \"kubernetes.io/projected/c1cf6bdd-85db-4669-a694-ac7ab7445792-kube-api-access-th8gz\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.846993 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.846963    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-token\") pod \"e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") " pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:50.911765 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.911711    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh" podStartSLOduration=2.911695493 podStartE2EDuration="2.911695493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:50.909528923 +0000 UTC m=+678.350565421" watchObservedRunningTime="2026-11-02 01:43:50.911695493 +0000 UTC m=+678.352731989"
Nov 02 01:43:50.945790 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:50.945739    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:51.034974 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:51.031104    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw" podStartSLOduration=2.03108747 podStartE2EDuration="2.03108747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:51.029793026 +0000 UTC m=+678.470829524" watchObservedRunningTime="2026-11-02 01:43:51.03108747 +0000 UTC m=+678.472123969"
Nov 02 01:43:51.037569 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:51.037027    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:51.051280 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:51.051252    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:51.322583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:51.322523    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"]
Nov 02 01:43:51.325057 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:43:51.325018    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1cf6bdd_85db_4669_a694_ac7ab7445792.slice/crio-bad5b920aaf617bf9b2a8871da4c08550ad279d6ac05ff202ffba1361e7a6b88 WatchSource:0}: Error finding container bad5b920aaf617bf9b2a8871da4c08550ad279d6ac05ff202ffba1361e7a6b88: Status 404 returned error can't find the container with id bad5b920aaf617bf9b2a8871da4c08550ad279d6ac05ff202ffba1361e7a6b88
Nov 02 01:43:51.326275 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:51.326253    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:51.328074 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:51.328039    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:43:51.328164 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:51.328116    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:43:51.328164 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:51.328151    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:43:51.332435 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:51.332417    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:51.796773 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:51.796740    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4" event={"ID":"c1cf6bdd-85db-4669-a694-ac7ab7445792","Type":"ContainerStarted","Data":"495f8716113a233c9208a5110f72df2e0f7a3ab94db54ffb77a9e59bb3591d5b"}
Nov 02 01:43:51.796773 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:51.796790    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4" event={"ID":"c1cf6bdd-85db-4669-a694-ac7ab7445792","Type":"ContainerStarted","Data":"bad5b920aaf617bf9b2a8871da4c08550ad279d6ac05ff202ffba1361e7a6b88"}
Nov 02 01:43:51.797281 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:51.797257    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:51.797413 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:51.797306    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:51.798054 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:51.798034    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:43:51.798202 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:51.798189    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:43:51.919435 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:51.919386    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4" podStartSLOduration=1.919371435 podStartE2EDuration="1.919371435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:51.917665102 +0000 UTC m=+679.358701601" watchObservedRunningTime="2026-11-02 01:43:51.919371435 +0000 UTC m=+679.360407931"
Nov 02 01:43:51.946050 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:51.946021    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:52.951694 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:52.951660    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:53.814210 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:53.814174    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:53.816576 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:53.816547    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:43:58.278598 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:58.278554    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4"]
Nov 02 01:43:58.283613 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:58.283588    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4"
Nov 02 01:43:58.329767 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:58.329740    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4"]
Nov 02 01:43:58.359704 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:58.359669    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e0047bd-4ecb-46b0-80ef-5c40b882dda6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4\" (UID: \"0e0047bd-4ecb-46b0-80ef-5c40b882dda6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4"
Nov 02 01:43:58.359857 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:58.359779    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67rn6\" (UniqueName: \"kubernetes.io/projected/0e0047bd-4ecb-46b0-80ef-5c40b882dda6-kube-api-access-67rn6\") pod \"payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4\" (UID: \"0e0047bd-4ecb-46b0-80ef-5c40b882dda6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4"
Nov 02 01:43:58.376552 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:58.376523    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-9470a3-dockercfg-tdjdh\""
Nov 02 01:43:58.376552 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:58.376523    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-9470a3\""
Nov 02 01:43:58.461200 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:58.461165    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67rn6\" (UniqueName: \"kubernetes.io/projected/0e0047bd-4ecb-46b0-80ef-5c40b882dda6-kube-api-access-67rn6\") pod \"payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4\" (UID: \"0e0047bd-4ecb-46b0-80ef-5c40b882dda6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4"
Nov 02 01:43:58.461370 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:58.461326    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e0047bd-4ecb-46b0-80ef-5c40b882dda6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4\" (UID: \"0e0047bd-4ecb-46b0-80ef-5c40b882dda6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4"
Nov 02 01:43:58.461928 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:58.461909    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e0047bd-4ecb-46b0-80ef-5c40b882dda6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4\" (UID: \"0e0047bd-4ecb-46b0-80ef-5c40b882dda6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4"
Nov 02 01:43:58.521807 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:58.521778    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67rn6\" (UniqueName: \"kubernetes.io/projected/0e0047bd-4ecb-46b0-80ef-5c40b882dda6-kube-api-access-67rn6\") pod \"payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4\" (UID: \"0e0047bd-4ecb-46b0-80ef-5c40b882dda6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4"
Nov 02 01:43:58.596950 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:58.596869    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4"
Nov 02 01:43:58.972534 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:58.972440    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4"]
Nov 02 01:43:58.974889 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:43:58.974859    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e0047bd_4ecb_46b0_80ef_5c40b882dda6.slice/crio-96df780073e845e42e7b7092c3f56a5a7e4b1dd1d376bfa880f9ca4e8f55910e WatchSource:0}: Error finding container 96df780073e845e42e7b7092c3f56a5a7e4b1dd1d376bfa880f9ca4e8f55910e: Status 404 returned error can't find the container with id 96df780073e845e42e7b7092c3f56a5a7e4b1dd1d376bfa880f9ca4e8f55910e
Nov 02 01:43:59.848494 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:43:59.848463    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4" event={"ID":"0e0047bd-4ecb-46b0-80ef-5c40b882dda6","Type":"ContainerStarted","Data":"96df780073e845e42e7b7092c3f56a5a7e4b1dd1d376bfa880f9ca4e8f55910e"}
Nov 02 01:44:01.861688 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:01.861654    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4" event={"ID":"0e0047bd-4ecb-46b0-80ef-5c40b882dda6","Type":"ContainerStarted","Data":"e212ac312bb499c523a96887e02592bc7e07f0a2e5f56dc9d899e18783bfcfe2"}
Nov 02 01:44:01.862084 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:01.861720    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4"
Nov 02 01:44:01.982822 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:01.982763    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4" podStartSLOduration=1.334905008 podStartE2EDuration="3.982745798s" totalImagesPullingTime="2.64784079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:01.981067757 +0000 UTC m=+689.422104254" watchObservedRunningTime="2026-11-02 01:44:01.982745798 +0000 UTC m=+689.423782340"
Nov 02 01:44:03.210380 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:03.210349    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7"]
Nov 02 01:44:03.215062 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:03.215042    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7"
Nov 02 01:44:03.267623 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:03.267597    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-ed0600\""
Nov 02 01:44:03.314459 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:03.314423    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c003811-1913-436c-b5c2-5e67bc7b72a1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7\" (UID: \"0c003811-1913-436c-b5c2-5e67bc7b72a1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7"
Nov 02 01:44:03.314620 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:03.314478    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmpg4\" (UniqueName: \"kubernetes.io/projected/0c003811-1913-436c-b5c2-5e67bc7b72a1-kube-api-access-gmpg4\") pod \"payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7\" (UID: \"0c003811-1913-436c-b5c2-5e67bc7b72a1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7"
Nov 02 01:44:03.316707 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:03.316683    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-ed0600-dockercfg-fdv5k\""
Nov 02 01:44:03.318743 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:03.318721    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7"]
Nov 02 01:44:03.415869 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:03.415841    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c003811-1913-436c-b5c2-5e67bc7b72a1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7\" (UID: \"0c003811-1913-436c-b5c2-5e67bc7b72a1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7"
Nov 02 01:44:03.415986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:03.415887    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmpg4\" (UniqueName: \"kubernetes.io/projected/0c003811-1913-436c-b5c2-5e67bc7b72a1-kube-api-access-gmpg4\") pod \"payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7\" (UID: \"0c003811-1913-436c-b5c2-5e67bc7b72a1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7"
Nov 02 01:44:03.416633 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:03.416610    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c003811-1913-436c-b5c2-5e67bc7b72a1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7\" (UID: \"0c003811-1913-436c-b5c2-5e67bc7b72a1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7"
Nov 02 01:44:03.476997 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:03.476933    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmpg4\" (UniqueName: \"kubernetes.io/projected/0c003811-1913-436c-b5c2-5e67bc7b72a1-kube-api-access-gmpg4\") pod \"payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7\" (UID: \"0c003811-1913-436c-b5c2-5e67bc7b72a1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7"
Nov 02 01:44:03.528392 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:03.528365    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7"
Nov 02 01:44:03.900350 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:03.900326    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7"]
Nov 02 01:44:03.902443 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:44:03.902417    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c003811_1913_436c_b5c2_5e67bc7b72a1.slice/crio-b4a34bf158f1f959cf0c26eec86d066588c57d91cdf83e53a33ef517745c2481 WatchSource:0}: Error finding container b4a34bf158f1f959cf0c26eec86d066588c57d91cdf83e53a33ef517745c2481: Status 404 returned error can't find the container with id b4a34bf158f1f959cf0c26eec86d066588c57d91cdf83e53a33ef517745c2481
Nov 02 01:44:04.876883 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:04.876848    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7" event={"ID":"0c003811-1913-436c-b5c2-5e67bc7b72a1","Type":"ContainerStarted","Data":"4e7d61ecb78d274040844a90e86dc13a0279f791eefcf0d099e0bde3b58bff12"}
Nov 02 01:44:04.877307 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:04.876906    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7" event={"ID":"0c003811-1913-436c-b5c2-5e67bc7b72a1","Type":"ContainerStarted","Data":"b4a34bf158f1f959cf0c26eec86d066588c57d91cdf83e53a33ef517745c2481"}
Nov 02 01:44:04.877307 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:04.876941    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7"
Nov 02 01:44:04.997810 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:04.997757    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7" podStartSLOduration=1.997741572 podStartE2EDuration="1.997741572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:04.995703486 +0000 UTC m=+692.436739984" watchObservedRunningTime="2026-11-02 01:44:04.997741572 +0000 UTC m=+692.438778068"
Nov 02 01:44:05.757744 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:05.757711    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8"]
Nov 02 01:44:05.761471 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:05.761451    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8"
Nov 02 01:44:05.813553 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:05.813531    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-0ebdb3\""
Nov 02 01:44:05.813687 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:05.813563    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-0ebdb3-dockercfg-7ckk2\""
Nov 02 01:44:05.844118 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:05.844088    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/450879f9-c443-4a35-a13d-6872b0294337-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8\" (UID: \"450879f9-c443-4a35-a13d-6872b0294337\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8"
Nov 02 01:44:05.844326 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:05.844145    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc42f\" (UniqueName: \"kubernetes.io/projected/450879f9-c443-4a35-a13d-6872b0294337-kube-api-access-tc42f\") pod \"payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8\" (UID: \"450879f9-c443-4a35-a13d-6872b0294337\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8"
Nov 02 01:44:05.864963 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:05.864935    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8"]
Nov 02 01:44:05.945895 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:05.945857    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/450879f9-c443-4a35-a13d-6872b0294337-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8\" (UID: \"450879f9-c443-4a35-a13d-6872b0294337\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8"
Nov 02 01:44:05.946322 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:05.945947    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc42f\" (UniqueName: \"kubernetes.io/projected/450879f9-c443-4a35-a13d-6872b0294337-kube-api-access-tc42f\") pod \"payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8\" (UID: \"450879f9-c443-4a35-a13d-6872b0294337\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8"
Nov 02 01:44:05.946699 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:05.946678    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/450879f9-c443-4a35-a13d-6872b0294337-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8\" (UID: \"450879f9-c443-4a35-a13d-6872b0294337\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8"
Nov 02 01:44:06.007315 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:06.007281    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc42f\" (UniqueName: \"kubernetes.io/projected/450879f9-c443-4a35-a13d-6872b0294337-kube-api-access-tc42f\") pod \"payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8\" (UID: \"450879f9-c443-4a35-a13d-6872b0294337\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8"
Nov 02 01:44:06.074518 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:06.074429    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8"
Nov 02 01:44:06.449295 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:06.449270    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8"]
Nov 02 01:44:06.451567 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:44:06.451539    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod450879f9_c443_4a35_a13d_6872b0294337.slice/crio-c4a973270438ed2d2c23c5481f687967102924b3fd0ce403d397fcd86f1fc2ec WatchSource:0}: Error finding container c4a973270438ed2d2c23c5481f687967102924b3fd0ce403d397fcd86f1fc2ec: Status 404 returned error can't find the container with id c4a973270438ed2d2c23c5481f687967102924b3fd0ce403d397fcd86f1fc2ec
Nov 02 01:44:06.891104 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:06.891070    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8" event={"ID":"450879f9-c443-4a35-a13d-6872b0294337","Type":"ContainerStarted","Data":"a5f046ab3d2daf6d0c5aee11e6a6b80ec0a2a733e4d4e5a8377c888bb54b37b0"}
Nov 02 01:44:06.891321 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:06.891127    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8" event={"ID":"450879f9-c443-4a35-a13d-6872b0294337","Type":"ContainerStarted","Data":"c4a973270438ed2d2c23c5481f687967102924b3fd0ce403d397fcd86f1fc2ec"}
Nov 02 01:44:06.891321 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:06.891258    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8"
Nov 02 01:44:07.007338 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:07.007286    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8" podStartSLOduration=2.007267357 podStartE2EDuration="2.007267357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:07.006534929 +0000 UTC m=+694.447571429" watchObservedRunningTime="2026-11-02 01:44:07.007267357 +0000 UTC m=+694.448303854"
Nov 02 01:44:11.318400 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:44:11.318357    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:44:12.798184 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:12.798149    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:44:12.869171 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:12.869146    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4"
Nov 02 01:44:12.960404 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:12.960372    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh"]
Nov 02 01:44:12.965667 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:12.965640    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh"
Nov 02 01:44:13.024878 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:13.024844    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-061fab-dockercfg-hqmkq\""
Nov 02 01:44:13.024878 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:13.024844    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-061fab\""
Nov 02 01:44:13.026006 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:13.025981    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh"]
Nov 02 01:44:13.126745 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:13.126671    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e89df719-7936-4331-980a-f2cbdfc0a8a4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh\" (UID: \"e89df719-7936-4331-980a-f2cbdfc0a8a4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh"
Nov 02 01:44:13.126745 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:13.126702    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7n7w\" (UniqueName: \"kubernetes.io/projected/e89df719-7936-4331-980a-f2cbdfc0a8a4-kube-api-access-b7n7w\") pod \"payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh\" (UID: \"e89df719-7936-4331-980a-f2cbdfc0a8a4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh"
Nov 02 01:44:13.227762 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:13.227733    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e89df719-7936-4331-980a-f2cbdfc0a8a4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh\" (UID: \"e89df719-7936-4331-980a-f2cbdfc0a8a4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh"
Nov 02 01:44:13.227762 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:13.227763    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7n7w\" (UniqueName: \"kubernetes.io/projected/e89df719-7936-4331-980a-f2cbdfc0a8a4-kube-api-access-b7n7w\") pod \"payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh\" (UID: \"e89df719-7936-4331-980a-f2cbdfc0a8a4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh"
Nov 02 01:44:13.228502 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:13.228478    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e89df719-7936-4331-980a-f2cbdfc0a8a4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh\" (UID: \"e89df719-7936-4331-980a-f2cbdfc0a8a4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh"
Nov 02 01:44:13.288911 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:13.288879    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7n7w\" (UniqueName: \"kubernetes.io/projected/e89df719-7936-4331-980a-f2cbdfc0a8a4-kube-api-access-b7n7w\") pod \"payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh\" (UID: \"e89df719-7936-4331-980a-f2cbdfc0a8a4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh"
Nov 02 01:44:13.292730 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:13.292708    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh"
Nov 02 01:44:13.667604 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:13.667552    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh"]
Nov 02 01:44:13.670022 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:44:13.669994    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode89df719_7936_4331_980a_f2cbdfc0a8a4.slice/crio-d21e713f0ec91d68150a9ed597c81e67406bb3f8c0546503c604352a218e1d18 WatchSource:0}: Error finding container d21e713f0ec91d68150a9ed597c81e67406bb3f8c0546503c604352a218e1d18: Status 404 returned error can't find the container with id d21e713f0ec91d68150a9ed597c81e67406bb3f8c0546503c604352a218e1d18
Nov 02 01:44:13.927760 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:13.927682    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh" event={"ID":"e89df719-7936-4331-980a-f2cbdfc0a8a4","Type":"ContainerStarted","Data":"436b84b98829467bf07c102fc2c0a8f3917744faada743cf5c086319571833c6"}
Nov 02 01:44:13.927760 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:13.927732    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh" event={"ID":"e89df719-7936-4331-980a-f2cbdfc0a8a4","Type":"ContainerStarted","Data":"d21e713f0ec91d68150a9ed597c81e67406bb3f8c0546503c604352a218e1d18"}
Nov 02 01:44:13.928184 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:13.927766    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh"
Nov 02 01:44:14.047990 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:14.047933    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh" podStartSLOduration=2.047918227 podStartE2EDuration="2.047918227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:14.045930973 +0000 UTC m=+701.486967470" watchObservedRunningTime="2026-11-02 01:44:14.047918227 +0000 UTC m=+701.488954724"
Nov 02 01:44:15.887575 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:15.887547    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7"
Nov 02 01:44:17.899403 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:17.899372    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8"
Nov 02 01:44:20.129278 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:20.129242    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr"]
Nov 02 01:44:20.133945 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:20.133923    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr"
Nov 02 01:44:20.190938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:20.190903    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr"]
Nov 02 01:44:20.247453 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:20.247421    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-c62671\""
Nov 02 01:44:20.247656 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:20.247542    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-c62671-dockercfg-chjxt\""
Nov 02 01:44:20.295565 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:20.295536    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822"]
Nov 02 01:44:20.299477 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:20.299459    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822"
Nov 02 01:44:20.312881 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:20.312856    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70879f63-ab1e-41ba-83e8-4df35dc40d1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr\" (UID: \"70879f63-ab1e-41ba-83e8-4df35dc40d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr"
Nov 02 01:44:20.313002 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:20.312906    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9g4k\" (UniqueName: \"kubernetes.io/projected/70879f63-ab1e-41ba-83e8-4df35dc40d1d-kube-api-access-w9g4k\") pod \"payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr\" (UID: \"70879f63-ab1e-41ba-83e8-4df35dc40d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr"
Nov 02 01:44:20.357066 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:20.357041    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822"]
Nov 02 01:44:20.413887 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:20.413816    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c6117a-dbee-4707-ba61-2e6971c1d52f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-c62671-656c54b597-zs822\" (UID: \"90c6117a-dbee-4707-ba61-2e6971c1d52f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822"
Nov 02 01:44:20.413887 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:20.413857    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70879f63-ab1e-41ba-83e8-4df35dc40d1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr\" (UID: \"70879f63-ab1e-41ba-83e8-4df35dc40d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr"
Nov 02 01:44:20.414054 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:20.413943    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9g4k\" (UniqueName: \"kubernetes.io/projected/70879f63-ab1e-41ba-83e8-4df35dc40d1d-kube-api-access-w9g4k\") pod \"payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr\" (UID: \"70879f63-ab1e-41ba-83e8-4df35dc40d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr"
Nov 02 01:44:20.414054 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:20.414012    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztvnp\" (UniqueName: \"kubernetes.io/projected/90c6117a-dbee-4707-ba61-2e6971c1d52f-kube-api-access-ztvnp\") pod \"payload-processing-e2e-worker-w2-c62671-656c54b597-zs822\" (UID: \"90c6117a-dbee-4707-ba61-2e6971c1d52f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822"
Nov 02 01:44:20.414562 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:20.414541    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70879f63-ab1e-41ba-83e8-4df35dc40d1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr\" (UID: \"70879f63-ab1e-41ba-83e8-4df35dc40d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr"
Nov 02 01:44:20.474793 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:20.474766    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9g4k\" (UniqueName: \"kubernetes.io/projected/70879f63-ab1e-41ba-83e8-4df35dc40d1d-kube-api-access-w9g4k\") pod \"payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr\" (UID: \"70879f63-ab1e-41ba-83e8-4df35dc40d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr"
Nov 02 01:44:20.515460 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:20.515430    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztvnp\" (UniqueName: \"kubernetes.io/projected/90c6117a-dbee-4707-ba61-2e6971c1d52f-kube-api-access-ztvnp\") pod \"payload-processing-e2e-worker-w2-c62671-656c54b597-zs822\" (UID: \"90c6117a-dbee-4707-ba61-2e6971c1d52f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822"
Nov 02 01:44:20.515564 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:20.515511    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c6117a-dbee-4707-ba61-2e6971c1d52f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-c62671-656c54b597-zs822\" (UID: \"90c6117a-dbee-4707-ba61-2e6971c1d52f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822"
Nov 02 01:44:20.516143 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:20.516127    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c6117a-dbee-4707-ba61-2e6971c1d52f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-c62671-656c54b597-zs822\" (UID: \"90c6117a-dbee-4707-ba61-2e6971c1d52f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822"
Nov 02 01:44:20.575704 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:20.575675    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztvnp\" (UniqueName: \"kubernetes.io/projected/90c6117a-dbee-4707-ba61-2e6971c1d52f-kube-api-access-ztvnp\") pod \"payload-processing-e2e-worker-w2-c62671-656c54b597-zs822\" (UID: \"90c6117a-dbee-4707-ba61-2e6971c1d52f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822"
Nov 02 01:44:20.614696 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:20.614670    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822"
Nov 02 01:44:20.746606 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:20.746580    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr"
Nov 02 01:44:20.988761 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:20.988733    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822"]
Nov 02 01:44:20.990528 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:44:20.990501    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c6117a_dbee_4707_ba61_2e6971c1d52f.slice/crio-1714c83b7d23ebc631ff430ffaa14743cc42ae3732836e7bfec1335b3a3cd382 WatchSource:0}: Error finding container 1714c83b7d23ebc631ff430ffaa14743cc42ae3732836e7bfec1335b3a3cd382: Status 404 returned error can't find the container with id 1714c83b7d23ebc631ff430ffaa14743cc42ae3732836e7bfec1335b3a3cd382
Nov 02 01:44:21.119498 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:21.119473    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr"]
Nov 02 01:44:21.122124 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:44:21.122100    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70879f63_ab1e_41ba_83e8_4df35dc40d1d.slice/crio-bc256ee29d8a2a559bfe70e9128c19a53ad048272f81b540ec0d152f24f776f3 WatchSource:0}: Error finding container bc256ee29d8a2a559bfe70e9128c19a53ad048272f81b540ec0d152f24f776f3: Status 404 returned error can't find the container with id bc256ee29d8a2a559bfe70e9128c19a53ad048272f81b540ec0d152f24f776f3
Nov 02 01:44:21.974395 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:21.974362    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr" event={"ID":"70879f63-ab1e-41ba-83e8-4df35dc40d1d","Type":"ContainerStarted","Data":"97c2c7497e33e3cf304f688087cc7e18c1005318b4710764e956e215a399a8ac"}
Nov 02 01:44:21.974873 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:21.974423    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr" event={"ID":"70879f63-ab1e-41ba-83e8-4df35dc40d1d","Type":"ContainerStarted","Data":"bc256ee29d8a2a559bfe70e9128c19a53ad048272f81b540ec0d152f24f776f3"}
Nov 02 01:44:21.974873 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:21.974452    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr"
Nov 02 01:44:21.975958 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:21.975936    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822" event={"ID":"90c6117a-dbee-4707-ba61-2e6971c1d52f","Type":"ContainerStarted","Data":"7afe8cd641902e780ff2e240c8e0d2e12b12d59bb147494b10618c3bda4f9ecb"}
Nov 02 01:44:21.976021 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:21.975974    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822" event={"ID":"90c6117a-dbee-4707-ba61-2e6971c1d52f","Type":"ContainerStarted","Data":"1714c83b7d23ebc631ff430ffaa14743cc42ae3732836e7bfec1335b3a3cd382"}
Nov 02 01:44:21.976021 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:21.976001    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822"
Nov 02 01:44:22.090730 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:22.090681    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr" podStartSLOduration=2.090665396 podStartE2EDuration="2.090665396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:22.08966559 +0000 UTC m=+709.530702088" watchObservedRunningTime="2026-11-02 01:44:22.090665396 +0000 UTC m=+709.531701895"
Nov 02 01:44:22.206518 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:22.206455    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822" podStartSLOduration=2.206440593 podStartE2EDuration="2.206440593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:22.204396954 +0000 UTC m=+709.645433453" watchObservedRunningTime="2026-11-02 01:44:22.206440593 +0000 UTC m=+709.647477090"
Nov 02 01:44:22.409845 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:22.409773    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:44:22.670107 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:22.670034    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg"]
Nov 02 01:44:22.675084 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:22.675065    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg"
Nov 02 01:44:22.728903 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:22.728861    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-dbe849-dockercfg-qmm9r\""
Nov 02 01:44:22.729844 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:22.729825    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-dbe849\""
Nov 02 01:44:22.731953 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:22.731925    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg"]
Nov 02 01:44:22.837490 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:22.837446    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mjzt\" (UniqueName: \"kubernetes.io/projected/89e5e146-c748-4d7f-aa22-4a77851f8817-kube-api-access-7mjzt\") pod \"payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg\" (UID: \"89e5e146-c748-4d7f-aa22-4a77851f8817\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg"
Nov 02 01:44:22.837671 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:22.837641    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89e5e146-c748-4d7f-aa22-4a77851f8817-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg\" (UID: \"89e5e146-c748-4d7f-aa22-4a77851f8817\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg"
Nov 02 01:44:22.938970 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:22.938876    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89e5e146-c748-4d7f-aa22-4a77851f8817-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg\" (UID: \"89e5e146-c748-4d7f-aa22-4a77851f8817\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg"
Nov 02 01:44:22.939129 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:22.939016    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mjzt\" (UniqueName: \"kubernetes.io/projected/89e5e146-c748-4d7f-aa22-4a77851f8817-kube-api-access-7mjzt\") pod \"payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg\" (UID: \"89e5e146-c748-4d7f-aa22-4a77851f8817\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg"
Nov 02 01:44:22.939659 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:22.939637    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89e5e146-c748-4d7f-aa22-4a77851f8817-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg\" (UID: \"89e5e146-c748-4d7f-aa22-4a77851f8817\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg"
Nov 02 01:44:22.999557 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:22.999516    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mjzt\" (UniqueName: \"kubernetes.io/projected/89e5e146-c748-4d7f-aa22-4a77851f8817-kube-api-access-7mjzt\") pod \"payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg\" (UID: \"89e5e146-c748-4d7f-aa22-4a77851f8817\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg"
Nov 02 01:44:23.289567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:23.289535    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg"
Nov 02 01:44:23.666101 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:23.666078    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg"]
Nov 02 01:44:23.668250 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:44:23.668203    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e5e146_c748_4d7f_aa22_4a77851f8817.slice/crio-e29679262f94f2a0c56d6359d8ba3beb0110c17be4bbcbcb9e9582259750e592 WatchSource:0}: Error finding container e29679262f94f2a0c56d6359d8ba3beb0110c17be4bbcbcb9e9582259750e592: Status 404 returned error can't find the container with id e29679262f94f2a0c56d6359d8ba3beb0110c17be4bbcbcb9e9582259750e592
Nov 02 01:44:23.989691 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:23.989648    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg" event={"ID":"89e5e146-c748-4d7f-aa22-4a77851f8817","Type":"ContainerStarted","Data":"099057a2dad0e178985208dfc3898b04a499917efd20c80ca09b23343ee1e446"}
Nov 02 01:44:23.989859 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:23.989715    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg" event={"ID":"89e5e146-c748-4d7f-aa22-4a77851f8817","Type":"ContainerStarted","Data":"e29679262f94f2a0c56d6359d8ba3beb0110c17be4bbcbcb9e9582259750e592"}
Nov 02 01:44:23.989859 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:23.989744    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg"
Nov 02 01:44:24.111248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:24.107504    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg" podStartSLOduration=2.107483768 podStartE2EDuration="2.107483768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:24.106590148 +0000 UTC m=+711.547626685" watchObservedRunningTime="2026-11-02 01:44:24.107483768 +0000 UTC m=+711.548520268"
Nov 02 01:44:24.935754 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:24.935719    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh"
Nov 02 01:44:31.693152 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:31.693117    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:44:32.985947 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:32.985921    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr"
Nov 02 01:44:32.986378 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:32.986096    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822"
Nov 02 01:44:33.067438 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:33.067408    5875 generic.go:356] "Generic (PLEG): container finished" podID="c1cf6bdd-85db-4669-a694-ac7ab7445792" containerID="495f8716113a233c9208a5110f72df2e0f7a3ab94db54ffb77a9e59bb3591d5b" exitCode=0
Nov 02 01:44:33.067600 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:33.067478    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4" event={"ID":"c1cf6bdd-85db-4669-a694-ac7ab7445792","Type":"ContainerDied","Data":"495f8716113a233c9208a5110f72df2e0f7a3ab94db54ffb77a9e59bb3591d5b"}
Nov 02 01:44:33.067852 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:33.067838    5875 scope.go:122] "RemoveContainer" containerID="495f8716113a233c9208a5110f72df2e0f7a3ab94db54ffb77a9e59bb3591d5b"
Nov 02 01:44:33.069147 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:33.069115    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:44:33.069309 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:33.069190    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:44:33.069309 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:33.069259    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:44:34.076031 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:34.075998    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4" event={"ID":"c1cf6bdd-85db-4669-a694-ac7ab7445792","Type":"ContainerStarted","Data":"0664e37c3b29a85f87d000542e574c4a421728e5134e5d944353248286603fb2"}
Nov 02 01:44:34.088830 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:34.088800    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz"]
Nov 02 01:44:34.092730 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:34.092711    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz"
Nov 02 01:44:34.144783 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:34.144751    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:44:34.144957 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:34.144815    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-061fab-serving-cert\""
Nov 02 01:44:34.144957 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:34.144847    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:44:34.177909 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:34.177881    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 01:44:34.178063 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:34.177952    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2n4c5\""
Nov 02 01:44:34.179373 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:34.179357    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1dfbd34-3b3f-477c-994d-6bb41f538eef-metrics-certs\") pod \"maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz\" (UID: \"c1dfbd34-3b3f-477c-994d-6bb41f538eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz"
Nov 02 01:44:34.179535 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:34.179511    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1dfbd34-3b3f-477c-994d-6bb41f538eef-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz\" (UID: \"c1dfbd34-3b3f-477c-994d-6bb41f538eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz"
Nov 02 01:44:34.179673 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:34.179636    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbpqv\" (UniqueName: \"kubernetes.io/projected/c1dfbd34-3b3f-477c-994d-6bb41f538eef-kube-api-access-wbpqv\") pod \"maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz\" (UID: \"c1dfbd34-3b3f-477c-994d-6bb41f538eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz"
Nov 02 01:44:34.179734 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:34.179699    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1dfbd34-3b3f-477c-994d-6bb41f538eef-maas-api-tls\") pod \"maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz\" (UID: \"c1dfbd34-3b3f-477c-994d-6bb41f538eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz"
Nov 02 01:44:34.180031 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:34.180010    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz"]
Nov 02 01:44:34.280734 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:34.280701    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1dfbd34-3b3f-477c-994d-6bb41f538eef-metrics-certs\") pod \"maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz\" (UID: \"c1dfbd34-3b3f-477c-994d-6bb41f538eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz"
Nov 02 01:44:34.280919 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:34.280762    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1dfbd34-3b3f-477c-994d-6bb41f538eef-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz\" (UID: \"c1dfbd34-3b3f-477c-994d-6bb41f538eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz"
Nov 02 01:44:34.280919 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:34.280797    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbpqv\" (UniqueName: \"kubernetes.io/projected/c1dfbd34-3b3f-477c-994d-6bb41f538eef-kube-api-access-wbpqv\") pod \"maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz\" (UID: \"c1dfbd34-3b3f-477c-994d-6bb41f538eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz"
Nov 02 01:44:34.280919 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:34.280828    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1dfbd34-3b3f-477c-994d-6bb41f538eef-maas-api-tls\") pod \"maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz\" (UID: \"c1dfbd34-3b3f-477c-994d-6bb41f538eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz"
Nov 02 01:44:34.281103 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:44:34.280961    5875 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-serving-cert: secret "maas-api-e2e-worker-w3-061fab-serving-cert" not found
Nov 02 01:44:34.281103 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:44:34.281031    5875 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1dfbd34-3b3f-477c-994d-6bb41f538eef-maas-api-tls podName:c1dfbd34-3b3f-477c-994d-6bb41f538eef nodeName:}" failed. No retries permitted until 2026-11-02 01:44:34.781015403 +0000 UTC m=+722.222051878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c1dfbd34-3b3f-477c-994d-6bb41f538eef-maas-api-tls") pod "maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz" (UID: "c1dfbd34-3b3f-477c-994d-6bb41f538eef") : secret "maas-api-e2e-worker-w3-061fab-serving-cert" not found
Nov 02 01:44:34.281527 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:34.281508    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1dfbd34-3b3f-477c-994d-6bb41f538eef-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz\" (UID: \"c1dfbd34-3b3f-477c-994d-6bb41f538eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz"
Nov 02 01:44:34.283311 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:34.283289    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1dfbd34-3b3f-477c-994d-6bb41f538eef-metrics-certs\") pod \"maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz\" (UID: \"c1dfbd34-3b3f-477c-994d-6bb41f538eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz"
Nov 02 01:44:34.349256 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:34.349142    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbpqv\" (UniqueName: \"kubernetes.io/projected/c1dfbd34-3b3f-477c-994d-6bb41f538eef-kube-api-access-wbpqv\") pod \"maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz\" (UID: \"c1dfbd34-3b3f-477c-994d-6bb41f538eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz"
Nov 02 01:44:34.787431 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:34.787394    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1dfbd34-3b3f-477c-994d-6bb41f538eef-maas-api-tls\") pod \"maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz\" (UID: \"c1dfbd34-3b3f-477c-994d-6bb41f538eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz"
Nov 02 01:44:34.790027 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:34.790005    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1dfbd34-3b3f-477c-994d-6bb41f538eef-maas-api-tls\") pod \"maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz\" (UID: \"c1dfbd34-3b3f-477c-994d-6bb41f538eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz"
Nov 02 01:44:34.946046 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:34.946013    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:44:34.947501 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:34.947470    5875 patch_prober.go:29] interesting pod/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 02 01:44:34.947630 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:34.947530    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4" podUID="c1cf6bdd-85db-4669-a694-ac7ab7445792" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 02 01:44:35.018248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:35.013390    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz"
Nov 02 01:44:35.022450 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:35.022425    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg"
Nov 02 01:44:35.395342 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:35.395309    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz"]
Nov 02 01:44:35.396376 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:44:35.396351    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1dfbd34_3b3f_477c_994d_6bb41f538eef.slice/crio-48b40494f21ce45628dc7d1de596f34432eb104d54f3476ab5b43127742dd39b WatchSource:0}: Error finding container 48b40494f21ce45628dc7d1de596f34432eb104d54f3476ab5b43127742dd39b: Status 404 returned error can't find the container with id 48b40494f21ce45628dc7d1de596f34432eb104d54f3476ab5b43127742dd39b
Nov 02 01:44:35.946531 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:35.946494    5875 patch_prober.go:29] interesting pod/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 02 01:44:35.946783 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:35.946552    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4" podUID="c1cf6bdd-85db-4669-a694-ac7ab7445792" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 02 01:44:36.091342 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:36.091311    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz" event={"ID":"c1dfbd34-3b3f-477c-994d-6bb41f538eef","Type":"ContainerStarted","Data":"48b40494f21ce45628dc7d1de596f34432eb104d54f3476ab5b43127742dd39b"}
Nov 02 01:44:36.946589 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:36.946545    5875 patch_prober.go:29] interesting pod/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 02 01:44:36.946977 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:36.946612    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4" podUID="c1cf6bdd-85db-4669-a694-ac7ab7445792" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 02 01:44:37.946788 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:37.946701    5875 patch_prober.go:29] interesting pod/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 02 01:44:37.946788 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:37.946761    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4" podUID="c1cf6bdd-85db-4669-a694-ac7ab7445792" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 02 01:44:38.162430 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:38.162391    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz" event={"ID":"c1dfbd34-3b3f-477c-994d-6bb41f538eef","Type":"ContainerStarted","Data":"84f6b62ee142d4f3ec7ffbbaea9a2fdd6794debe61089678e47f415dad87f681"}
Nov 02 01:44:38.162618 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:38.162538    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz"
Nov 02 01:44:38.288576 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:38.288515    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz" podStartSLOduration=2.06205918 podStartE2EDuration="4.288500614s" totalImagesPullingTime="2.226441434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:38.288458119 +0000 UTC m=+725.729494618" watchObservedRunningTime="2026-11-02 01:44:38.288500614 +0000 UTC m=+725.729537110"
Nov 02 01:44:38.818246 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:38.817327    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:44:38.946260 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:38.946208    5875 patch_prober.go:29] interesting pod/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 02 01:44:38.946427 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:38.946276    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4" podUID="c1cf6bdd-85db-4669-a694-ac7ab7445792" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 02 01:44:39.950553 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:39.950526    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:44:39.951455 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:39.951438    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 01:44:41.343885 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:44:41.343839    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:44:44.174846 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:44.174820    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz"
Nov 02 01:44:50.265122 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:50.265046    5875 generic.go:356] "Generic (PLEG): container finished" podID="166343e8-524c-4668-9094-a503991be4f9" containerID="8f3f00f580c3ccd7ee044a5da972c67d9186da7c6a00b8cb8c07d52dc9b8b51d" exitCode=0
Nov 02 01:44:50.265580 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:50.265124    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw" event={"ID":"166343e8-524c-4668-9094-a503991be4f9","Type":"ContainerDied","Data":"8f3f00f580c3ccd7ee044a5da972c67d9186da7c6a00b8cb8c07d52dc9b8b51d"}
Nov 02 01:44:50.265580 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:50.265546    5875 scope.go:122] "RemoveContainer" containerID="8f3f00f580c3ccd7ee044a5da972c67d9186da7c6a00b8cb8c07d52dc9b8b51d"
Nov 02 01:44:50.266981 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:50.266949    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:44:50.267072 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:50.267018    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:44:50.267072 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:50.267046    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:44:51.272377 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:51.272342    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw" event={"ID":"166343e8-524c-4668-9094-a503991be4f9","Type":"ContainerStarted","Data":"e363896987f9192d328bf8d611238d41fa218712e4fa3794af11dbb506c6843a"}
Nov 02 01:44:51.274049 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:51.274024    5875 generic.go:356] "Generic (PLEG): container finished" podID="05f7e0b6-dc17-4891-9817-242be8465c54" containerID="295e19acbb53caf69ec7f0f20cf688103e7b889e62d320dc8c7e525a17c1f33d" exitCode=0
Nov 02 01:44:51.274250 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:51.274101    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh" event={"ID":"05f7e0b6-dc17-4891-9817-242be8465c54","Type":"ContainerDied","Data":"295e19acbb53caf69ec7f0f20cf688103e7b889e62d320dc8c7e525a17c1f33d"}
Nov 02 01:44:51.274481 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:51.274467    5875 scope.go:122] "RemoveContainer" containerID="295e19acbb53caf69ec7f0f20cf688103e7b889e62d320dc8c7e525a17c1f33d"
Nov 02 01:44:51.275695 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:51.275669    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:44:51.275760 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:51.275734    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:44:51.275798 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:51.275766    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:44:51.802248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:51.801315    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:44:51.814246 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:51.807316    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:44:52.041252 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:52.036592    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:44:52.041252 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:52.038843    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 02 01:44:52.041252 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:52.038897    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw" podUID="166343e8-524c-4668-9094-a503991be4f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 02 01:44:52.275360 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:52.275327    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:44:52.282268 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:52.282237    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh" event={"ID":"05f7e0b6-dc17-4891-9817-242be8465c54","Type":"ContainerStarted","Data":"7dcf8805adcd0c037b18a35a45dfc5dc6a190a503be98f61da4dfe2a49e2891b"}
Nov 02 01:44:52.333256 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:52.332711    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:44:52.333256 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:52.332836    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 02 01:44:52.333256 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:52.332878    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh" podUID="05f7e0b6-dc17-4891-9817-242be8465c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 02 01:44:53.051660 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:53.051632    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 02 01:44:53.051889 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:53.051869    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw" podUID="166343e8-524c-4668-9094-a503991be4f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 02 01:44:53.338305 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:53.337324    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 02 01:44:53.338305 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:53.337413    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh" podUID="05f7e0b6-dc17-4891-9817-242be8465c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 02 01:44:54.040334 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:54.040300    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 02 01:44:54.040614 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:54.040588    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw" podUID="166343e8-524c-4668-9094-a503991be4f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 02 01:44:54.327303 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:54.326539    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 02 01:44:54.327303 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:54.326612    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh" podUID="05f7e0b6-dc17-4891-9817-242be8465c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 02 01:44:55.057248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:55.050415    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 02 01:44:55.057248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:55.050477    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw" podUID="166343e8-524c-4668-9094-a503991be4f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 02 01:44:55.333303 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:55.331542    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 02 01:44:55.333303 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:55.331599    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh" podUID="05f7e0b6-dc17-4891-9817-242be8465c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 02 01:44:56.036653 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:56.036615    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 02 01:44:56.036953 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:56.036927    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw" podUID="166343e8-524c-4668-9094-a503991be4f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 02 01:44:56.327858 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:56.327758    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 02 01:44:56.328384 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:56.328304    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh" podUID="05f7e0b6-dc17-4891-9817-242be8465c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 02 01:44:57.041136 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:57.041102    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:44:57.042133 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:57.042112    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 01:44:57.325788 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:57.325705    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 02 01:44:57.325788 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:57.325757    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh" podUID="05f7e0b6-dc17-4891-9817-242be8465c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 02 01:44:58.330802 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:58.330772    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:44:58.331213 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:58.331052    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:44:58.331717 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:44:58.331702    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 01:45:00.168556 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.168520    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv"]
Nov 02 01:45:00.174770 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.174745    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z"]
Nov 02 01:45:00.174949 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.174893    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv"
Nov 02 01:45:00.179293 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.179272    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr"]
Nov 02 01:45:00.179412 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.179393    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z"
Nov 02 01:45:00.183190 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.183175    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr"
Nov 02 01:45:00.207086 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.207065    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjs6x\" (UniqueName: \"kubernetes.io/projected/d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3-kube-api-access-qjs6x\") pod \"maas-api-key-cleanup-29893065-4xwsr\" (UID: \"d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr"
Nov 02 01:45:00.207190 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.207105    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f87fae2-7ed3-4fdd-b222-52de762decad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv\" (UID: \"5f87fae2-7ed3-4fdd-b222-52de762decad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv"
Nov 02 01:45:00.207250 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.207196    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-687n9\" (UniqueName: \"kubernetes.io/projected/b44c667e-e039-4f1b-a3dc-c4b5dd9d1320-kube-api-access-687n9\") pod \"maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z\" (UID: \"b44c667e-e039-4f1b-a3dc-c4b5dd9d1320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z"
Nov 02 01:45:00.207305 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.207285    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b44c667e-e039-4f1b-a3dc-c4b5dd9d1320-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z\" (UID: \"b44c667e-e039-4f1b-a3dc-c4b5dd9d1320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z"
Nov 02 01:45:00.207342 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.207332    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893065-4xwsr\" (UID: \"d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr"
Nov 02 01:45:00.207400 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.207385    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh7tb\" (UniqueName: \"kubernetes.io/projected/5f87fae2-7ed3-4fdd-b222-52de762decad-kube-api-access-gh7tb\") pod \"maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv\" (UID: \"5f87fae2-7ed3-4fdd-b222-52de762decad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv"
Nov 02 01:45:00.218013 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.217994    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds"]
Nov 02 01:45:00.222076 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.222063    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds"
Nov 02 01:45:00.227287 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.227266    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:45:00.271516 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.271488    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv"]
Nov 02 01:45:00.277784 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.277758    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr"]
Nov 02 01:45:00.279599 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.279574    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z"]
Nov 02 01:45:00.310631 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.308241    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-687n9\" (UniqueName: \"kubernetes.io/projected/b44c667e-e039-4f1b-a3dc-c4b5dd9d1320-kube-api-access-687n9\") pod \"maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z\" (UID: \"b44c667e-e039-4f1b-a3dc-c4b5dd9d1320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z"
Nov 02 01:45:00.310631 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.308313    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b44c667e-e039-4f1b-a3dc-c4b5dd9d1320-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z\" (UID: \"b44c667e-e039-4f1b-a3dc-c4b5dd9d1320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z"
Nov 02 01:45:00.310631 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.308362    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893065-4xwsr\" (UID: \"d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr"
Nov 02 01:45:00.310631 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.308402    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh7tb\" (UniqueName: \"kubernetes.io/projected/5f87fae2-7ed3-4fdd-b222-52de762decad-kube-api-access-gh7tb\") pod \"maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv\" (UID: \"5f87fae2-7ed3-4fdd-b222-52de762decad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv"
Nov 02 01:45:00.310631 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.308500    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d063b2-202d-4ccf-b60d-7a594e6e125e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds\" (UID: \"41d063b2-202d-4ccf-b60d-7a594e6e125e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds"
Nov 02 01:45:00.310631 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.308556    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjs6x\" (UniqueName: \"kubernetes.io/projected/d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3-kube-api-access-qjs6x\") pod \"maas-api-key-cleanup-29893065-4xwsr\" (UID: \"d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr"
Nov 02 01:45:00.310631 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.308604    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f87fae2-7ed3-4fdd-b222-52de762decad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv\" (UID: \"5f87fae2-7ed3-4fdd-b222-52de762decad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv"
Nov 02 01:45:00.310631 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.308641    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwxxq\" (UniqueName: \"kubernetes.io/projected/41d063b2-202d-4ccf-b60d-7a594e6e125e-kube-api-access-vwxxq\") pod \"maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds\" (UID: \"41d063b2-202d-4ccf-b60d-7a594e6e125e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds"
Nov 02 01:45:00.310631 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.309573    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b44c667e-e039-4f1b-a3dc-c4b5dd9d1320-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z\" (UID: \"b44c667e-e039-4f1b-a3dc-c4b5dd9d1320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z"
Nov 02 01:45:00.310631 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.309993    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893065-4xwsr\" (UID: \"d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr"
Nov 02 01:45:00.310631 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.310485    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f87fae2-7ed3-4fdd-b222-52de762decad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv\" (UID: \"5f87fae2-7ed3-4fdd-b222-52de762decad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv"
Nov 02 01:45:00.369731 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.369698    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-687n9\" (UniqueName: \"kubernetes.io/projected/b44c667e-e039-4f1b-a3dc-c4b5dd9d1320-kube-api-access-687n9\") pod \"maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z\" (UID: \"b44c667e-e039-4f1b-a3dc-c4b5dd9d1320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z"
Nov 02 01:45:00.369880 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.369772    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjs6x\" (UniqueName: \"kubernetes.io/projected/d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3-kube-api-access-qjs6x\") pod \"maas-api-key-cleanup-29893065-4xwsr\" (UID: \"d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr"
Nov 02 01:45:00.369880 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.369786    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh7tb\" (UniqueName: \"kubernetes.io/projected/5f87fae2-7ed3-4fdd-b222-52de762decad-kube-api-access-gh7tb\") pod \"maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv\" (UID: \"5f87fae2-7ed3-4fdd-b222-52de762decad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv"
Nov 02 01:45:00.389004 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.388878    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds"]
Nov 02 01:45:00.411457 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.411423    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d063b2-202d-4ccf-b60d-7a594e6e125e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds\" (UID: \"41d063b2-202d-4ccf-b60d-7a594e6e125e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds"
Nov 02 01:45:00.411657 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.411510    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwxxq\" (UniqueName: \"kubernetes.io/projected/41d063b2-202d-4ccf-b60d-7a594e6e125e-kube-api-access-vwxxq\") pod \"maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds\" (UID: \"41d063b2-202d-4ccf-b60d-7a594e6e125e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds"
Nov 02 01:45:00.412638 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.412611    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d063b2-202d-4ccf-b60d-7a594e6e125e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds\" (UID: \"41d063b2-202d-4ccf-b60d-7a594e6e125e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds"
Nov 02 01:45:00.472643 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.472614    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwxxq\" (UniqueName: \"kubernetes.io/projected/41d063b2-202d-4ccf-b60d-7a594e6e125e-kube-api-access-vwxxq\") pod \"maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds\" (UID: \"41d063b2-202d-4ccf-b60d-7a594e6e125e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds"
Nov 02 01:45:00.490427 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.490403    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv"
Nov 02 01:45:00.498762 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.498743    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z"
Nov 02 01:45:00.504659 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.504635    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr"
Nov 02 01:45:00.535599 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.535569    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds"
Nov 02 01:45:00.875492 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.875472    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv"]
Nov 02 01:45:00.877987 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:45:00.877960    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f87fae2_7ed3_4fdd_b222_52de762decad.slice/crio-3a051f5d0670e04423fef383bc1ff36243225ce65d3b2dd2610d5b9f6bcf8259 WatchSource:0}: Error finding container 3a051f5d0670e04423fef383bc1ff36243225ce65d3b2dd2610d5b9f6bcf8259: Status 404 returned error can't find the container with id 3a051f5d0670e04423fef383bc1ff36243225ce65d3b2dd2610d5b9f6bcf8259
Nov 02 01:45:00.924984 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.924950    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr"]
Nov 02 01:45:00.926780 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:45:00.926756    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f3afd6_0d79_4dd1_be14_bcc3c2337ac3.slice/crio-492d3a8031651492bc990a5c45af941a05ee8fd2f0977233d355ca133d7354f3 WatchSource:0}: Error finding container 492d3a8031651492bc990a5c45af941a05ee8fd2f0977233d355ca133d7354f3: Status 404 returned error can't find the container with id 492d3a8031651492bc990a5c45af941a05ee8fd2f0977233d355ca133d7354f3
Nov 02 01:45:00.927931 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.927886    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z"]
Nov 02 01:45:00.929034 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:45:00.929014    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44c667e_e039_4f1b_a3dc_c4b5dd9d1320.slice/crio-044da03c75305f517c9b0248b88e86abef08b442fe84d9e718f656cb7c7e2f76 WatchSource:0}: Error finding container 044da03c75305f517c9b0248b88e86abef08b442fe84d9e718f656cb7c7e2f76: Status 404 returned error can't find the container with id 044da03c75305f517c9b0248b88e86abef08b442fe84d9e718f656cb7c7e2f76
Nov 02 01:45:00.975894 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:00.975877    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds"]
Nov 02 01:45:00.977844 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:45:00.977817    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d063b2_202d_4ccf_b60d_7a594e6e125e.slice/crio-39942495473e61b7f38cf11c4efb0ace484e3d52728f9ede39124c48e1bc7868 WatchSource:0}: Error finding container 39942495473e61b7f38cf11c4efb0ace484e3d52728f9ede39124c48e1bc7868: Status 404 returned error can't find the container with id 39942495473e61b7f38cf11c4efb0ace484e3d52728f9ede39124c48e1bc7868
Nov 02 01:45:01.367330 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:01.367257    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr" event={"ID":"d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3","Type":"ContainerStarted","Data":"492d3a8031651492bc990a5c45af941a05ee8fd2f0977233d355ca133d7354f3"}
Nov 02 01:45:01.368665 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:01.368642    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv" event={"ID":"5f87fae2-7ed3-4fdd-b222-52de762decad","Type":"ContainerStarted","Data":"3a051f5d0670e04423fef383bc1ff36243225ce65d3b2dd2610d5b9f6bcf8259"}
Nov 02 01:45:01.369965 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:01.369936    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds" event={"ID":"41d063b2-202d-4ccf-b60d-7a594e6e125e","Type":"ContainerStarted","Data":"39942495473e61b7f38cf11c4efb0ace484e3d52728f9ede39124c48e1bc7868"}
Nov 02 01:45:01.371147 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:01.371126    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z" event={"ID":"b44c667e-e039-4f1b-a3dc-c4b5dd9d1320","Type":"ContainerStarted","Data":"044da03c75305f517c9b0248b88e86abef08b442fe84d9e718f656cb7c7e2f76"}
Nov 02 01:45:02.104508 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:02.104479    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:45:06.413897 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:06.413798    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr" event={"ID":"d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3","Type":"ContainerStarted","Data":"fa36e849f5a323ec2f85d7d5b75d6c08686ba1ef6e34866c1e2293c0722a76fd"}
Nov 02 01:45:06.415448 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:06.415421    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv" event={"ID":"5f87fae2-7ed3-4fdd-b222-52de762decad","Type":"ContainerStarted","Data":"c2bfb308c6ce8eefad7099b780bea1504823027374757627858b7f1402c68a78"}
Nov 02 01:45:06.417006 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:06.416978    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds" event={"ID":"41d063b2-202d-4ccf-b60d-7a594e6e125e","Type":"ContainerStarted","Data":"757892c1f30e932dc8cfcd4ca74eb245232d37386ff368c4c587a1f69d97488f"}
Nov 02 01:45:06.418419 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:06.418396    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z" event={"ID":"b44c667e-e039-4f1b-a3dc-c4b5dd9d1320","Type":"ContainerStarted","Data":"40f149975d8bb37ef7e835ed5c2e62452ab1c109eb91bee9c66d6254ebd803ad"}
Nov 02 01:45:06.533500 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:06.533451    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr" podStartSLOduration=1.630659404 podStartE2EDuration="6.533437206s" totalImagesPullingTime="4.902777802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:06.531903672 +0000 UTC m=+753.972940160" watchObservedRunningTime="2026-11-02 01:45:06.533437206 +0000 UTC m=+753.974473702"
Nov 02 01:45:06.649278 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:06.649213    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv" podStartSLOduration=1.7008835709999999 podStartE2EDuration="6.649199545s" totalImagesPullingTime="4.948315974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:06.647130544 +0000 UTC m=+754.088167042" watchObservedRunningTime="2026-11-02 01:45:06.649199545 +0000 UTC m=+754.090236221"
Nov 02 01:45:06.761349 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:06.761300    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds" podStartSLOduration=2.116135311 podStartE2EDuration="6.761288246s" totalImagesPullingTime="4.645152935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:06.760647973 +0000 UTC m=+754.201684471" watchObservedRunningTime="2026-11-02 01:45:06.761288246 +0000 UTC m=+754.202324744"
Nov 02 01:45:06.887576 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:06.887524    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z" podStartSLOduration=2.245056467 podStartE2EDuration="6.887513068s" totalImagesPullingTime="4.642456601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:06.88600567 +0000 UTC m=+754.327042168" watchObservedRunningTime="2026-11-02 01:45:06.887513068 +0000 UTC m=+754.328549566"
Nov 02 01:45:07.427500 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:07.427468    5875 generic.go:356] "Generic (PLEG): container finished" podID="d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3" containerID="fa36e849f5a323ec2f85d7d5b75d6c08686ba1ef6e34866c1e2293c0722a76fd" exitCode=60
Nov 02 01:45:07.427968 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:07.427598    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr" event={"ID":"d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3","Type":"ContainerDied","Data":"fa36e849f5a323ec2f85d7d5b75d6c08686ba1ef6e34866c1e2293c0722a76fd"}
Nov 02 01:45:07.427968 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:07.427824    5875 scope.go:122] "RemoveContainer" containerID="fa36e849f5a323ec2f85d7d5b75d6c08686ba1ef6e34866c1e2293c0722a76fd"
Nov 02 01:45:07.429591 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:07.429563    5875 generic.go:356] "Generic (PLEG): container finished" podID="5f87fae2-7ed3-4fdd-b222-52de762decad" containerID="c2bfb308c6ce8eefad7099b780bea1504823027374757627858b7f1402c68a78" exitCode=60
Nov 02 01:45:07.429712 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:07.429654    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv" event={"ID":"5f87fae2-7ed3-4fdd-b222-52de762decad","Type":"ContainerDied","Data":"c2bfb308c6ce8eefad7099b780bea1504823027374757627858b7f1402c68a78"}
Nov 02 01:45:07.430135 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:07.429975    5875 scope.go:122] "RemoveContainer" containerID="c2bfb308c6ce8eefad7099b780bea1504823027374757627858b7f1402c68a78"
Nov 02 01:45:07.432015 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:07.431990    5875 generic.go:356] "Generic (PLEG): container finished" podID="41d063b2-202d-4ccf-b60d-7a594e6e125e" containerID="757892c1f30e932dc8cfcd4ca74eb245232d37386ff368c4c587a1f69d97488f" exitCode=60
Nov 02 01:45:07.432405 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:07.432068    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds" event={"ID":"41d063b2-202d-4ccf-b60d-7a594e6e125e","Type":"ContainerDied","Data":"757892c1f30e932dc8cfcd4ca74eb245232d37386ff368c4c587a1f69d97488f"}
Nov 02 01:45:07.432508 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:07.432281    5875 scope.go:122] "RemoveContainer" containerID="757892c1f30e932dc8cfcd4ca74eb245232d37386ff368c4c587a1f69d97488f"
Nov 02 01:45:07.434019 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:07.433991    5875 generic.go:356] "Generic (PLEG): container finished" podID="b44c667e-e039-4f1b-a3dc-c4b5dd9d1320" containerID="40f149975d8bb37ef7e835ed5c2e62452ab1c109eb91bee9c66d6254ebd803ad" exitCode=60
Nov 02 01:45:07.434121 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:07.434030    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z" event={"ID":"b44c667e-e039-4f1b-a3dc-c4b5dd9d1320","Type":"ContainerDied","Data":"40f149975d8bb37ef7e835ed5c2e62452ab1c109eb91bee9c66d6254ebd803ad"}
Nov 02 01:45:07.434623 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:07.434610    5875 scope.go:122] "RemoveContainer" containerID="40f149975d8bb37ef7e835ed5c2e62452ab1c109eb91bee9c66d6254ebd803ad"
Nov 02 01:45:08.442090 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:08.442060    5875 generic.go:356] "Generic (PLEG): container finished" podID="d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3" containerID="7f616fc4043c6746b7516cbda1745189327c6825f65ed10ac7a44883f47675b4" exitCode=60
Nov 02 01:45:08.442510 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:08.442158    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr" event={"ID":"d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3","Type":"ContainerDied","Data":"7f616fc4043c6746b7516cbda1745189327c6825f65ed10ac7a44883f47675b4"}
Nov 02 01:45:08.442510 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:08.442246    5875 scope.go:122] "RemoveContainer" containerID="fa36e849f5a323ec2f85d7d5b75d6c08686ba1ef6e34866c1e2293c0722a76fd"
Nov 02 01:45:08.442510 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:08.442435    5875 scope.go:122] "RemoveContainer" containerID="7f616fc4043c6746b7516cbda1745189327c6825f65ed10ac7a44883f47675b4"
Nov 02 01:45:08.442699 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:45:08.442681    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893065-4xwsr_odh-ai-gateway-infra(d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr" podUID="d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3"
Nov 02 01:45:08.444267 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:08.444243    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv" event={"ID":"5f87fae2-7ed3-4fdd-b222-52de762decad","Type":"ContainerStarted","Data":"3d916031869961a459adaad030d457e8f15a9c7a3c332b16f37064a26d59ec2e"}
Nov 02 01:45:08.446086 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:08.446064    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds" event={"ID":"41d063b2-202d-4ccf-b60d-7a594e6e125e","Type":"ContainerStarted","Data":"4f173ceda4e51fce3689122fbcdd3275110e6eb9cec9fcf3cc0f0907f15fc396"}
Nov 02 01:45:08.448065 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:08.448040    5875 generic.go:356] "Generic (PLEG): container finished" podID="b44c667e-e039-4f1b-a3dc-c4b5dd9d1320" containerID="dd7416570ada62ec773dd74977499d68ca703e7cf81f298abd2dd3b47d501026" exitCode=60
Nov 02 01:45:08.448160 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:08.448117    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z" event={"ID":"b44c667e-e039-4f1b-a3dc-c4b5dd9d1320","Type":"ContainerDied","Data":"dd7416570ada62ec773dd74977499d68ca703e7cf81f298abd2dd3b47d501026"}
Nov 02 01:45:08.448454 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:08.448439    5875 scope.go:122] "RemoveContainer" containerID="dd7416570ada62ec773dd74977499d68ca703e7cf81f298abd2dd3b47d501026"
Nov 02 01:45:08.448709 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:45:08.448686    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z_odh-ai-gateway-infra(b44c667e-e039-4f1b-a3dc-c4b5dd9d1320)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z" podUID="b44c667e-e039-4f1b-a3dc-c4b5dd9d1320"
Nov 02 01:45:08.454482 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:08.454466    5875 scope.go:122] "RemoveContainer" containerID="40f149975d8bb37ef7e835ed5c2e62452ab1c109eb91bee9c66d6254ebd803ad"
Nov 02 01:45:09.456366 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:09.456332    5875 scope.go:122] "RemoveContainer" containerID="7f616fc4043c6746b7516cbda1745189327c6825f65ed10ac7a44883f47675b4"
Nov 02 01:45:09.456824 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:45:09.456574    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893065-4xwsr_odh-ai-gateway-infra(d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr" podUID="d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3"
Nov 02 01:45:09.458086 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:09.458061    5875 generic.go:356] "Generic (PLEG): container finished" podID="5f87fae2-7ed3-4fdd-b222-52de762decad" containerID="3d916031869961a459adaad030d457e8f15a9c7a3c332b16f37064a26d59ec2e" exitCode=60
Nov 02 01:45:09.458201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:09.458141    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv" event={"ID":"5f87fae2-7ed3-4fdd-b222-52de762decad","Type":"ContainerDied","Data":"3d916031869961a459adaad030d457e8f15a9c7a3c332b16f37064a26d59ec2e"}
Nov 02 01:45:09.458296 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:09.458243    5875 scope.go:122] "RemoveContainer" containerID="c2bfb308c6ce8eefad7099b780bea1504823027374757627858b7f1402c68a78"
Nov 02 01:45:09.458349 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:09.458316    5875 scope.go:122] "RemoveContainer" containerID="3d916031869961a459adaad030d457e8f15a9c7a3c332b16f37064a26d59ec2e"
Nov 02 01:45:09.458571 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:45:09.458536    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv_odh-ai-gateway-infra(5f87fae2-7ed3-4fdd-b222-52de762decad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv" podUID="5f87fae2-7ed3-4fdd-b222-52de762decad"
Nov 02 01:45:09.460018 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:09.460000    5875 generic.go:356] "Generic (PLEG): container finished" podID="41d063b2-202d-4ccf-b60d-7a594e6e125e" containerID="4f173ceda4e51fce3689122fbcdd3275110e6eb9cec9fcf3cc0f0907f15fc396" exitCode=60
Nov 02 01:45:09.460281 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:09.460070    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds" event={"ID":"41d063b2-202d-4ccf-b60d-7a594e6e125e","Type":"ContainerDied","Data":"4f173ceda4e51fce3689122fbcdd3275110e6eb9cec9fcf3cc0f0907f15fc396"}
Nov 02 01:45:09.460281 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:09.460208    5875 scope.go:122] "RemoveContainer" containerID="4f173ceda4e51fce3689122fbcdd3275110e6eb9cec9fcf3cc0f0907f15fc396"
Nov 02 01:45:09.460430 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:45:09.460412    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds_odh-ai-gateway-infra(41d063b2-202d-4ccf-b60d-7a594e6e125e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds" podUID="41d063b2-202d-4ccf-b60d-7a594e6e125e"
Nov 02 01:45:09.461819 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:09.461802    5875 scope.go:122] "RemoveContainer" containerID="dd7416570ada62ec773dd74977499d68ca703e7cf81f298abd2dd3b47d501026"
Nov 02 01:45:09.462000 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:45:09.461976    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z_odh-ai-gateway-infra(b44c667e-e039-4f1b-a3dc-c4b5dd9d1320)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z" podUID="b44c667e-e039-4f1b-a3dc-c4b5dd9d1320"
Nov 02 01:45:09.470605 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:09.470589    5875 scope.go:122] "RemoveContainer" containerID="757892c1f30e932dc8cfcd4ca74eb245232d37386ff368c4c587a1f69d97488f"
Nov 02 01:45:10.470540 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:10.470509    5875 scope.go:122] "RemoveContainer" containerID="3d916031869961a459adaad030d457e8f15a9c7a3c332b16f37064a26d59ec2e"
Nov 02 01:45:10.470941 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:45:10.470781    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv_odh-ai-gateway-infra(5f87fae2-7ed3-4fdd-b222-52de762decad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv" podUID="5f87fae2-7ed3-4fdd-b222-52de762decad"
Nov 02 01:45:10.472157 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:10.472136    5875 scope.go:122] "RemoveContainer" containerID="4f173ceda4e51fce3689122fbcdd3275110e6eb9cec9fcf3cc0f0907f15fc396"
Nov 02 01:45:10.472376 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:45:10.472358    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds_odh-ai-gateway-infra(41d063b2-202d-4ccf-b60d-7a594e6e125e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds" podUID="41d063b2-202d-4ccf-b60d-7a594e6e125e"
Nov 02 01:45:11.318442 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:45:11.318407    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:45:20.687355 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:20.687322    5875 scope.go:122] "RemoveContainer" containerID="dd7416570ada62ec773dd74977499d68ca703e7cf81f298abd2dd3b47d501026"
Nov 02 01:45:20.687764 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:20.687465    5875 scope.go:122] "RemoveContainer" containerID="4f173ceda4e51fce3689122fbcdd3275110e6eb9cec9fcf3cc0f0907f15fc396"
Nov 02 01:45:21.538912 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:21.538881    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds" event={"ID":"41d063b2-202d-4ccf-b60d-7a594e6e125e","Type":"ContainerStarted","Data":"b4816b0b8123489696841bc16932c8c5214e896a695139788590da943cb62322"}
Nov 02 01:45:21.540800 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:21.540778    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z" event={"ID":"b44c667e-e039-4f1b-a3dc-c4b5dd9d1320","Type":"ContainerStarted","Data":"427c13fcda13733da21827f6bc6e21762d9540a6c53a8da0d2a5d95c6e2e6946"}
Nov 02 01:45:21.687195 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:21.687172    5875 scope.go:122] "RemoveContainer" containerID="3d916031869961a459adaad030d457e8f15a9c7a3c332b16f37064a26d59ec2e"
Nov 02 01:45:21.687415 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:21.687395    5875 scope.go:122] "RemoveContainer" containerID="7f616fc4043c6746b7516cbda1745189327c6825f65ed10ac7a44883f47675b4"
Nov 02 01:45:22.548152 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:22.548111    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv" event={"ID":"5f87fae2-7ed3-4fdd-b222-52de762decad","Type":"ContainerStarted","Data":"d425e45cff58474c682355e519d0a53f425b9123e9f724072f508d483815079a"}
Nov 02 01:45:22.550047 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:22.550023    5875 generic.go:356] "Generic (PLEG): container finished" podID="41d063b2-202d-4ccf-b60d-7a594e6e125e" containerID="b4816b0b8123489696841bc16932c8c5214e896a695139788590da943cb62322" exitCode=60
Nov 02 01:45:22.550170 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:22.550104    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds" event={"ID":"41d063b2-202d-4ccf-b60d-7a594e6e125e","Type":"ContainerDied","Data":"b4816b0b8123489696841bc16932c8c5214e896a695139788590da943cb62322"}
Nov 02 01:45:22.550245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:22.550173    5875 scope.go:122] "RemoveContainer" containerID="4f173ceda4e51fce3689122fbcdd3275110e6eb9cec9fcf3cc0f0907f15fc396"
Nov 02 01:45:22.550487 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:22.550473    5875 scope.go:122] "RemoveContainer" containerID="b4816b0b8123489696841bc16932c8c5214e896a695139788590da943cb62322"
Nov 02 01:45:22.550763 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:45:22.550740    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds_odh-ai-gateway-infra(41d063b2-202d-4ccf-b60d-7a594e6e125e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds" podUID="41d063b2-202d-4ccf-b60d-7a594e6e125e"
Nov 02 01:45:22.551966 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:22.551949    5875 generic.go:356] "Generic (PLEG): container finished" podID="b44c667e-e039-4f1b-a3dc-c4b5dd9d1320" containerID="427c13fcda13733da21827f6bc6e21762d9540a6c53a8da0d2a5d95c6e2e6946" exitCode=60
Nov 02 01:45:22.552060 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:22.552022    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z" event={"ID":"b44c667e-e039-4f1b-a3dc-c4b5dd9d1320","Type":"ContainerDied","Data":"427c13fcda13733da21827f6bc6e21762d9540a6c53a8da0d2a5d95c6e2e6946"}
Nov 02 01:45:22.552317 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:22.552301    5875 scope.go:122] "RemoveContainer" containerID="427c13fcda13733da21827f6bc6e21762d9540a6c53a8da0d2a5d95c6e2e6946"
Nov 02 01:45:22.552546 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:45:22.552519    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z_odh-ai-gateway-infra(b44c667e-e039-4f1b-a3dc-c4b5dd9d1320)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z" podUID="b44c667e-e039-4f1b-a3dc-c4b5dd9d1320"
Nov 02 01:45:22.553927 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:22.553911    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr" event={"ID":"d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3","Type":"ContainerStarted","Data":"50c1b19729489c611e2964af04de407fd04ac4a7cfee27453b81c516dacd4838"}
Nov 02 01:45:22.562204 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:22.562187    5875 scope.go:122] "RemoveContainer" containerID="dd7416570ada62ec773dd74977499d68ca703e7cf81f298abd2dd3b47d501026"
Nov 02 01:45:22.667586 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:22.667562    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds"]
Nov 02 01:45:22.777602 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:22.777565    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z"]
Nov 02 01:45:23.561527 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:23.561493    5875 generic.go:356] "Generic (PLEG): container finished" podID="d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3" containerID="50c1b19729489c611e2964af04de407fd04ac4a7cfee27453b81c516dacd4838" exitCode=60
Nov 02 01:45:23.561678 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:23.561561    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr" event={"ID":"d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3","Type":"ContainerDied","Data":"50c1b19729489c611e2964af04de407fd04ac4a7cfee27453b81c516dacd4838"}
Nov 02 01:45:23.561678 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:23.561625    5875 scope.go:122] "RemoveContainer" containerID="7f616fc4043c6746b7516cbda1745189327c6825f65ed10ac7a44883f47675b4"
Nov 02 01:45:23.561950 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:23.561935    5875 scope.go:122] "RemoveContainer" containerID="50c1b19729489c611e2964af04de407fd04ac4a7cfee27453b81c516dacd4838"
Nov 02 01:45:23.562258 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:45:23.562216    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893065-4xwsr_odh-ai-gateway-infra(d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr" podUID="d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3"
Nov 02 01:45:23.563639 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:23.563619    5875 generic.go:356] "Generic (PLEG): container finished" podID="5f87fae2-7ed3-4fdd-b222-52de762decad" containerID="d425e45cff58474c682355e519d0a53f425b9123e9f724072f508d483815079a" exitCode=60
Nov 02 01:45:23.563752 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:23.563700    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv" event={"ID":"5f87fae2-7ed3-4fdd-b222-52de762decad","Type":"ContainerDied","Data":"d425e45cff58474c682355e519d0a53f425b9123e9f724072f508d483815079a"}
Nov 02 01:45:23.564016 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:23.564002    5875 scope.go:122] "RemoveContainer" containerID="d425e45cff58474c682355e519d0a53f425b9123e9f724072f508d483815079a"
Nov 02 01:45:23.564259 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:45:23.564238    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv_odh-ai-gateway-infra(5f87fae2-7ed3-4fdd-b222-52de762decad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv" podUID="5f87fae2-7ed3-4fdd-b222-52de762decad"
Nov 02 01:45:23.574251 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:23.574204    5875 scope.go:122] "RemoveContainer" containerID="3d916031869961a459adaad030d457e8f15a9c7a3c332b16f37064a26d59ec2e"
Nov 02 01:45:23.674357 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:23.674322    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv"]
Nov 02 01:45:23.749786 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:23.749765    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds"
Nov 02 01:45:23.760306 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:23.760284    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z"
Nov 02 01:45:23.892160 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:23.892077    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41d063b2-202d-4ccf-b60d-7a594e6e125e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d063b2-202d-4ccf-b60d-7a594e6e125e-maas-api-service-ca\") pod \"41d063b2-202d-4ccf-b60d-7a594e6e125e\" (UID: \"41d063b2-202d-4ccf-b60d-7a594e6e125e\") "
Nov 02 01:45:23.892660 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:23.892164    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d063b2-202d-4ccf-b60d-7a594e6e125e-kube-api-access-vwxxq\" (UniqueName: \"kubernetes.io/projected/41d063b2-202d-4ccf-b60d-7a594e6e125e-kube-api-access-vwxxq\") pod \"41d063b2-202d-4ccf-b60d-7a594e6e125e\" (UID: \"41d063b2-202d-4ccf-b60d-7a594e6e125e\") "
Nov 02 01:45:23.892660 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:23.892205    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b44c667e-e039-4f1b-a3dc-c4b5dd9d1320-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b44c667e-e039-4f1b-a3dc-c4b5dd9d1320-maas-api-service-ca\") pod \"b44c667e-e039-4f1b-a3dc-c4b5dd9d1320\" (UID: \"b44c667e-e039-4f1b-a3dc-c4b5dd9d1320\") "
Nov 02 01:45:23.892660 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:23.892262    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b44c667e-e039-4f1b-a3dc-c4b5dd9d1320-kube-api-access-687n9\" (UniqueName: \"kubernetes.io/projected/b44c667e-e039-4f1b-a3dc-c4b5dd9d1320-kube-api-access-687n9\") pod \"b44c667e-e039-4f1b-a3dc-c4b5dd9d1320\" (UID: \"b44c667e-e039-4f1b-a3dc-c4b5dd9d1320\") "
Nov 02 01:45:23.892660 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:23.892539    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d063b2-202d-4ccf-b60d-7a594e6e125e-maas-api-service-ca" pod "41d063b2-202d-4ccf-b60d-7a594e6e125e" (UID: "41d063b2-202d-4ccf-b60d-7a594e6e125e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:23.892660 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:23.892593    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44c667e-e039-4f1b-a3dc-c4b5dd9d1320-maas-api-service-ca" pod "b44c667e-e039-4f1b-a3dc-c4b5dd9d1320" (UID: "b44c667e-e039-4f1b-a3dc-c4b5dd9d1320"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:23.892660 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:23.892657    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d063b2-202d-4ccf-b60d-7a594e6e125e-maas-api-service-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:23.894586 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:23.894568    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d063b2-202d-4ccf-b60d-7a594e6e125e-kube-api-access-vwxxq" pod "41d063b2-202d-4ccf-b60d-7a594e6e125e" (UID: "41d063b2-202d-4ccf-b60d-7a594e6e125e"). InnerVolumeSpecName "kube-api-access-vwxxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:23.894658 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:23.894596    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44c667e-e039-4f1b-a3dc-c4b5dd9d1320-kube-api-access-687n9" pod "b44c667e-e039-4f1b-a3dc-c4b5dd9d1320" (UID: "b44c667e-e039-4f1b-a3dc-c4b5dd9d1320"). InnerVolumeSpecName "kube-api-access-687n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:23.993660 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:23.993625    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwxxq\" (UniqueName: \"kubernetes.io/projected/41d063b2-202d-4ccf-b60d-7a594e6e125e-kube-api-access-vwxxq\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:23.993812 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:23.993663    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b44c667e-e039-4f1b-a3dc-c4b5dd9d1320-maas-api-service-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:23.993812 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:23.993682    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-687n9\" (UniqueName: \"kubernetes.io/projected/b44c667e-e039-4f1b-a3dc-c4b5dd9d1320-kube-api-access-687n9\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:24.028008 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:24.027952    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr"]
Nov 02 01:45:24.572973 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:24.572938    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:45:24.577884 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:24.577862    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds"
Nov 02 01:45:24.577884 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:24.577874    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds" event={"ID":"41d063b2-202d-4ccf-b60d-7a594e6e125e","Type":"ContainerDied","Data":"39942495473e61b7f38cf11c4efb0ace484e3d52728f9ede39124c48e1bc7868"}
Nov 02 01:45:24.578041 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:24.577933    5875 scope.go:122] "RemoveContainer" containerID="b4816b0b8123489696841bc16932c8c5214e896a695139788590da943cb62322"
Nov 02 01:45:24.579387 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:24.579365    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z"
Nov 02 01:45:24.579516 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:24.579362    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z" event={"ID":"b44c667e-e039-4f1b-a3dc-c4b5dd9d1320","Type":"ContainerDied","Data":"044da03c75305f517c9b0248b88e86abef08b442fe84d9e718f656cb7c7e2f76"}
Nov 02 01:45:24.588537 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:24.588503    5875 scope.go:122] "RemoveContainer" containerID="427c13fcda13733da21827f6bc6e21762d9540a6c53a8da0d2a5d95c6e2e6946"
Nov 02 01:45:24.727758 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:24.727626    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv"
Nov 02 01:45:24.766963 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:24.766938    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr"
Nov 02 01:45:24.809409 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:24.809374    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z"]
Nov 02 01:45:24.818911 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:24.818882    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9470a3-29893065-c5m5z"]
Nov 02 01:45:24.904061 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:24.903979    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f87fae2-7ed3-4fdd-b222-52de762decad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f87fae2-7ed3-4fdd-b222-52de762decad-maas-api-service-ca\") pod \"5f87fae2-7ed3-4fdd-b222-52de762decad\" (UID: \"5f87fae2-7ed3-4fdd-b222-52de762decad\") "
Nov 02 01:45:24.904701 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:24.904069    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3-maas-api-service-ca\") pod \"d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3\" (UID: \"d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3\") "
Nov 02 01:45:24.904701 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:24.904159    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3-kube-api-access-qjs6x\" (UniqueName: \"kubernetes.io/projected/d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3-kube-api-access-qjs6x\") pod \"d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3\" (UID: \"d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3\") "
Nov 02 01:45:24.904701 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:24.904203    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f87fae2-7ed3-4fdd-b222-52de762decad-kube-api-access-gh7tb\" (UniqueName: \"kubernetes.io/projected/5f87fae2-7ed3-4fdd-b222-52de762decad-kube-api-access-gh7tb\") pod \"5f87fae2-7ed3-4fdd-b222-52de762decad\" (UID: \"5f87fae2-7ed3-4fdd-b222-52de762decad\") "
Nov 02 01:45:24.904701 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:24.904342    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f87fae2-7ed3-4fdd-b222-52de762decad-maas-api-service-ca" pod "5f87fae2-7ed3-4fdd-b222-52de762decad" (UID: "5f87fae2-7ed3-4fdd-b222-52de762decad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:24.904701 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:24.904458    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3-maas-api-service-ca" pod "d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3" (UID: "d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:45:24.904701 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:24.904690    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f87fae2-7ed3-4fdd-b222-52de762decad-maas-api-service-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:24.905033 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:24.904712    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3-maas-api-service-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:24.906516 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:24.906483    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f87fae2-7ed3-4fdd-b222-52de762decad-kube-api-access-gh7tb" pod "5f87fae2-7ed3-4fdd-b222-52de762decad" (UID: "5f87fae2-7ed3-4fdd-b222-52de762decad"). InnerVolumeSpecName "kube-api-access-gh7tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:24.906638 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:24.906568    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3-kube-api-access-qjs6x" pod "d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3" (UID: "d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3"). InnerVolumeSpecName "kube-api-access-qjs6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:45:25.006195 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:25.006165    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjs6x\" (UniqueName: \"kubernetes.io/projected/d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3-kube-api-access-qjs6x\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:25.006195 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:25.006190    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh7tb\" (UniqueName: \"kubernetes.io/projected/5f87fae2-7ed3-4fdd-b222-52de762decad-kube-api-access-gh7tb\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:45:25.041957 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:25.041917    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds"]
Nov 02 01:45:25.047642 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:25.047615    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893065-8jqds"]
Nov 02 01:45:25.586370 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:25.586338    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv"
Nov 02 01:45:25.586568 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:25.586337    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv" event={"ID":"5f87fae2-7ed3-4fdd-b222-52de762decad","Type":"ContainerDied","Data":"3a051f5d0670e04423fef383bc1ff36243225ce65d3b2dd2610d5b9f6bcf8259"}
Nov 02 01:45:25.586568 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:25.586505    5875 scope.go:122] "RemoveContainer" containerID="d425e45cff58474c682355e519d0a53f425b9123e9f724072f508d483815079a"
Nov 02 01:45:25.590302 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:25.590266    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr" event={"ID":"d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3","Type":"ContainerDied","Data":"492d3a8031651492bc990a5c45af941a05ee8fd2f0977233d355ca133d7354f3"}
Nov 02 01:45:25.590396 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:25.590310    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr"
Nov 02 01:45:25.598102 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:25.598081    5875 scope.go:122] "RemoveContainer" containerID="50c1b19729489c611e2964af04de407fd04ac4a7cfee27453b81c516dacd4838"
Nov 02 01:45:25.750036 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:25.749950    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv"]
Nov 02 01:45:25.754796 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:25.754765    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-55bc84-29893065-5q2gv"]
Nov 02 01:45:25.908474 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:25.908388    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr"]
Nov 02 01:45:25.912773 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:25.912748    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893065-4xwsr"]
Nov 02 01:45:26.694856 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:26.694821    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41d063b2-202d-4ccf-b60d-7a594e6e125e" path="/var/lib/kubelet/pods/41d063b2-202d-4ccf-b60d-7a594e6e125e/volumes"
Nov 02 01:45:26.695312 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:26.695296    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f87fae2-7ed3-4fdd-b222-52de762decad" path="/var/lib/kubelet/pods/5f87fae2-7ed3-4fdd-b222-52de762decad/volumes"
Nov 02 01:45:26.695659 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:26.695646    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b44c667e-e039-4f1b-a3dc-c4b5dd9d1320" path="/var/lib/kubelet/pods/b44c667e-e039-4f1b-a3dc-c4b5dd9d1320/volumes"
Nov 02 01:45:26.696293 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:26.696279    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3" path="/var/lib/kubelet/pods/d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3/volumes"
Nov 02 01:45:41.318437 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:45:41.318398    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:45:41.873607 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:45:41.873573    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:46:09.542320 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:46:09.542288    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:46:11.319113 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:46:11.319021    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:46:35.228229 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:46:35.228181    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:46:41.318041 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:46:41.318002    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:47:03.738399 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:03.738362    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:47:04.603015 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.602980    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"]
Nov 02 01:47:04.603833 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.603812    5875 cpu_manager.go:414] "containerMap: removing container" podUID="5f87fae2-7ed3-4fdd-b222-52de762decad" containerName="cleanup"
Nov 02 01:47:04.603897 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.603839    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f87fae2-7ed3-4fdd-b222-52de762decad" containerName="cleanup"
Nov 02 01:47:04.603897 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.603856    5875 cpu_manager.go:414] "containerMap: removing container" podUID="41d063b2-202d-4ccf-b60d-7a594e6e125e" containerName="cleanup"
Nov 02 01:47:04.603897 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.603864    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d063b2-202d-4ccf-b60d-7a594e6e125e" containerName="cleanup"
Nov 02 01:47:04.603897 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.603877    5875 cpu_manager.go:414] "containerMap: removing container" podUID="41d063b2-202d-4ccf-b60d-7a594e6e125e" containerName="cleanup"
Nov 02 01:47:04.603897 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.603884    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d063b2-202d-4ccf-b60d-7a594e6e125e" containerName="cleanup"
Nov 02 01:47:04.603897 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.603895    5875 cpu_manager.go:414] "containerMap: removing container" podUID="b44c667e-e039-4f1b-a3dc-c4b5dd9d1320" containerName="cleanup"
Nov 02 01:47:04.604097 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.603902    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44c667e-e039-4f1b-a3dc-c4b5dd9d1320" containerName="cleanup"
Nov 02 01:47:04.604097 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.603913    5875 cpu_manager.go:414] "containerMap: removing container" podUID="5f87fae2-7ed3-4fdd-b222-52de762decad" containerName="cleanup"
Nov 02 01:47:04.604097 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.603920    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f87fae2-7ed3-4fdd-b222-52de762decad" containerName="cleanup"
Nov 02 01:47:04.604097 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.603951    5875 cpu_manager.go:414] "containerMap: removing container" podUID="d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3" containerName="cleanup"
Nov 02 01:47:04.604097 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.603958    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3" containerName="cleanup"
Nov 02 01:47:04.604097 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.603967    5875 cpu_manager.go:414] "containerMap: removing container" podUID="d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3" containerName="cleanup"
Nov 02 01:47:04.604097 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.603974    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3" containerName="cleanup"
Nov 02 01:47:04.604097 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.604006    5875 cpu_manager.go:414] "containerMap: removing container" podUID="b44c667e-e039-4f1b-a3dc-c4b5dd9d1320" containerName="cleanup"
Nov 02 01:47:04.604097 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.604013    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44c667e-e039-4f1b-a3dc-c4b5dd9d1320" containerName="cleanup"
Nov 02 01:47:04.604097 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.604031    5875 cpu_manager.go:414] "containerMap: removing container" podUID="b44c667e-e039-4f1b-a3dc-c4b5dd9d1320" containerName="cleanup"
Nov 02 01:47:04.604097 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.604038    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44c667e-e039-4f1b-a3dc-c4b5dd9d1320" containerName="cleanup"
Nov 02 01:47:04.604097 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.604053    5875 cpu_manager.go:414] "containerMap: removing container" podUID="5f87fae2-7ed3-4fdd-b222-52de762decad" containerName="cleanup"
Nov 02 01:47:04.604097 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.604060    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f87fae2-7ed3-4fdd-b222-52de762decad" containerName="cleanup"
Nov 02 01:47:04.604510 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.604179    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="b44c667e-e039-4f1b-a3dc-c4b5dd9d1320" containerName="cleanup"
Nov 02 01:47:04.604510 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.604194    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="b44c667e-e039-4f1b-a3dc-c4b5dd9d1320" containerName="cleanup"
Nov 02 01:47:04.604510 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.604203    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f87fae2-7ed3-4fdd-b222-52de762decad" containerName="cleanup"
Nov 02 01:47:04.604510 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.604214    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="41d063b2-202d-4ccf-b60d-7a594e6e125e" containerName="cleanup"
Nov 02 01:47:04.604510 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.604242    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3" containerName="cleanup"
Nov 02 01:47:04.604510 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.604252    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f87fae2-7ed3-4fdd-b222-52de762decad" containerName="cleanup"
Nov 02 01:47:04.604510 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.604260    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="41d063b2-202d-4ccf-b60d-7a594e6e125e" containerName="cleanup"
Nov 02 01:47:04.604510 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.604268    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="41d063b2-202d-4ccf-b60d-7a594e6e125e" containerName="cleanup"
Nov 02 01:47:04.604510 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.604278    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f87fae2-7ed3-4fdd-b222-52de762decad" containerName="cleanup"
Nov 02 01:47:04.604510 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.604291    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3" containerName="cleanup"
Nov 02 01:47:04.604510 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.604299    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3" containerName="cleanup"
Nov 02 01:47:04.608132 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.608113    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:47:04.694583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.694558    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e39fc3d-d730-4df7-8d10-2376656a88d2-tls-certs\") pod \"premium-w4-ed0600-kserve-788774b5dc-lzntx\" (UID: \"6e39fc3d-d730-4df7-8d10-2376656a88d2\") " pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:47:04.694743 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.694610    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8rkk\" (UniqueName: \"kubernetes.io/projected/6e39fc3d-d730-4df7-8d10-2376656a88d2-kube-api-access-b8rkk\") pod \"premium-w4-ed0600-kserve-788774b5dc-lzntx\" (UID: \"6e39fc3d-d730-4df7-8d10-2376656a88d2\") " pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:47:04.694743 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.694680    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-tmp-dir\") pod \"premium-w4-ed0600-kserve-788774b5dc-lzntx\" (UID: \"6e39fc3d-d730-4df7-8d10-2376656a88d2\") " pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:47:04.694743 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.694721    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-home\") pod \"premium-w4-ed0600-kserve-788774b5dc-lzntx\" (UID: \"6e39fc3d-d730-4df7-8d10-2376656a88d2\") " pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:47:04.694743 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.694742    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-dshm\") pod \"premium-w4-ed0600-kserve-788774b5dc-lzntx\" (UID: \"6e39fc3d-d730-4df7-8d10-2376656a88d2\") " pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:47:04.694877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.694772    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-model-cache\") pod \"premium-w4-ed0600-kserve-788774b5dc-lzntx\" (UID: \"6e39fc3d-d730-4df7-8d10-2376656a88d2\") " pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:47:04.711021 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.711001    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ed0600\"/\"openshift-service-ca.crt\""
Nov 02 01:47:04.711021 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.711013    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ed0600\"/\"default-dockercfg-qr9gk\""
Nov 02 01:47:04.711198 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.711029    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ed0600\"/\"kube-root-ca.crt\""
Nov 02 01:47:04.711198 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.711088    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ed0600\"/\"premium-w4-ed0600-kserve-self-signed-certs\""
Nov 02 01:47:04.713065 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.713048    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"]
Nov 02 01:47:04.796085 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.796053    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-dshm\") pod \"premium-w4-ed0600-kserve-788774b5dc-lzntx\" (UID: \"6e39fc3d-d730-4df7-8d10-2376656a88d2\") " pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:47:04.796085 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.796084    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-model-cache\") pod \"premium-w4-ed0600-kserve-788774b5dc-lzntx\" (UID: \"6e39fc3d-d730-4df7-8d10-2376656a88d2\") " pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:47:04.796574 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.796170    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e39fc3d-d730-4df7-8d10-2376656a88d2-tls-certs\") pod \"premium-w4-ed0600-kserve-788774b5dc-lzntx\" (UID: \"6e39fc3d-d730-4df7-8d10-2376656a88d2\") " pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:47:04.796574 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.796249    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8rkk\" (UniqueName: \"kubernetes.io/projected/6e39fc3d-d730-4df7-8d10-2376656a88d2-kube-api-access-b8rkk\") pod \"premium-w4-ed0600-kserve-788774b5dc-lzntx\" (UID: \"6e39fc3d-d730-4df7-8d10-2376656a88d2\") " pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:47:04.796574 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.796288    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-tmp-dir\") pod \"premium-w4-ed0600-kserve-788774b5dc-lzntx\" (UID: \"6e39fc3d-d730-4df7-8d10-2376656a88d2\") " pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:47:04.796574 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.796324    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-home\") pod \"premium-w4-ed0600-kserve-788774b5dc-lzntx\" (UID: \"6e39fc3d-d730-4df7-8d10-2376656a88d2\") " pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:47:04.796823 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.796587    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-model-cache\") pod \"premium-w4-ed0600-kserve-788774b5dc-lzntx\" (UID: \"6e39fc3d-d730-4df7-8d10-2376656a88d2\") " pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:47:04.796823 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.796618    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-home\") pod \"premium-w4-ed0600-kserve-788774b5dc-lzntx\" (UID: \"6e39fc3d-d730-4df7-8d10-2376656a88d2\") " pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:47:04.796823 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.796631    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-tmp-dir\") pod \"premium-w4-ed0600-kserve-788774b5dc-lzntx\" (UID: \"6e39fc3d-d730-4df7-8d10-2376656a88d2\") " pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:47:04.798553 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.798531    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-dshm\") pod \"premium-w4-ed0600-kserve-788774b5dc-lzntx\" (UID: \"6e39fc3d-d730-4df7-8d10-2376656a88d2\") " pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:47:04.798773 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.798756    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e39fc3d-d730-4df7-8d10-2376656a88d2-tls-certs\") pod \"premium-w4-ed0600-kserve-788774b5dc-lzntx\" (UID: \"6e39fc3d-d730-4df7-8d10-2376656a88d2\") " pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:47:04.857246 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.857174    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8rkk\" (UniqueName: \"kubernetes.io/projected/6e39fc3d-d730-4df7-8d10-2376656a88d2-kube-api-access-b8rkk\") pod \"premium-w4-ed0600-kserve-788774b5dc-lzntx\" (UID: \"6e39fc3d-d730-4df7-8d10-2376656a88d2\") " pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:47:04.923059 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:04.923035    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:47:05.323767 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:05.323738    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"]
Nov 02 01:47:05.325364 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:47:05.325337    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e39fc3d_d730_4df7_8d10_2376656a88d2.slice/crio-e27ca9db4263a819edf97972f350144c43ffedfcecf1e4aa72dd48afde24306e WatchSource:0}: Error finding container e27ca9db4263a819edf97972f350144c43ffedfcecf1e4aa72dd48afde24306e: Status 404 returned error can't find the container with id e27ca9db4263a819edf97972f350144c43ffedfcecf1e4aa72dd48afde24306e
Nov 02 01:47:06.154834 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:06.154800    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" event={"ID":"6e39fc3d-d730-4df7-8d10-2376656a88d2","Type":"ContainerStarted","Data":"9e8b080b794a240e71ef7100917d80541d46fe09759aa38a53cd1318ea031730"}
Nov 02 01:47:06.154834 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:06.154838    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" event={"ID":"6e39fc3d-d730-4df7-8d10-2376656a88d2","Type":"ContainerStarted","Data":"e27ca9db4263a819edf97972f350144c43ffedfcecf1e4aa72dd48afde24306e"}
Nov 02 01:47:06.274557 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:06.274500    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" podStartSLOduration=2.27448576 podStartE2EDuration="2.27448576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:47:06.272705304 +0000 UTC m=+873.713741804" watchObservedRunningTime="2026-11-02 01:47:06.27448576 +0000 UTC m=+873.715522256"
Nov 02 01:47:06.923519 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:06.923492    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:47:11.318643 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:47:11.318603    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:47:14.923523 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:14.923484    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:47:14.936170 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:14.936140    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:47:15.224635 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:15.224605    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:47:25.962896 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:25.962859    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:47:34.994266 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:34.994215    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"]
Nov 02 01:47:34.994950 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:34.994921    5875 cpu_manager.go:414] "containerMap: removing container" podUID="41d063b2-202d-4ccf-b60d-7a594e6e125e" containerName="cleanup"
Nov 02 01:47:34.995003 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:34.994953    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d063b2-202d-4ccf-b60d-7a594e6e125e" containerName="cleanup"
Nov 02 01:47:34.995003 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:34.994987    5875 cpu_manager.go:414] "containerMap: removing container" podUID="d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3" containerName="cleanup"
Nov 02 01:47:34.995003 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:34.994996    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f3afd6-0d79-4dd1-be14-bcc3c2337ac3" containerName="cleanup"
Nov 02 01:47:34.995140 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:34.995122    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="b44c667e-e039-4f1b-a3dc-c4b5dd9d1320" containerName="cleanup"
Nov 02 01:47:34.999109 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:34.999087    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:47:35.002844 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:35.002817    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-home\") pod \"premium-w3-061fab-kserve-7bb7d67d85-92kbb\" (UID: \"c9628178-6287-4ba3-9467-93f84ae883ee\") " pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:47:35.002978 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:35.002853    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-dshm\") pod \"premium-w3-061fab-kserve-7bb7d67d85-92kbb\" (UID: \"c9628178-6287-4ba3-9467-93f84ae883ee\") " pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:47:35.002978 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:35.002872    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-tmp-dir\") pod \"premium-w3-061fab-kserve-7bb7d67d85-92kbb\" (UID: \"c9628178-6287-4ba3-9467-93f84ae883ee\") " pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:47:35.002978 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:35.002887    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk9sv\" (UniqueName: \"kubernetes.io/projected/c9628178-6287-4ba3-9467-93f84ae883ee-kube-api-access-tk9sv\") pod \"premium-w3-061fab-kserve-7bb7d67d85-92kbb\" (UID: \"c9628178-6287-4ba3-9467-93f84ae883ee\") " pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:47:35.003101 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:35.003070    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-model-cache\") pod \"premium-w3-061fab-kserve-7bb7d67d85-92kbb\" (UID: \"c9628178-6287-4ba3-9467-93f84ae883ee\") " pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:47:35.003148 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:35.003127    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9628178-6287-4ba3-9467-93f84ae883ee-tls-certs\") pod \"premium-w3-061fab-kserve-7bb7d67d85-92kbb\" (UID: \"c9628178-6287-4ba3-9467-93f84ae883ee\") " pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:47:35.053057 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:35.053027    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-061fab\"/\"kube-root-ca.crt\""
Nov 02 01:47:35.102236 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:35.102200    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-061fab\"/\"openshift-service-ca.crt\""
Nov 02 01:47:35.104277 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:35.104209    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-home\") pod \"premium-w3-061fab-kserve-7bb7d67d85-92kbb\" (UID: \"c9628178-6287-4ba3-9467-93f84ae883ee\") " pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:47:35.104277 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:35.104271    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-dshm\") pod \"premium-w3-061fab-kserve-7bb7d67d85-92kbb\" (UID: \"c9628178-6287-4ba3-9467-93f84ae883ee\") " pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:47:35.104486 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:35.104297    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-tmp-dir\") pod \"premium-w3-061fab-kserve-7bb7d67d85-92kbb\" (UID: \"c9628178-6287-4ba3-9467-93f84ae883ee\") " pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:47:35.104486 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:35.104321    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk9sv\" (UniqueName: \"kubernetes.io/projected/c9628178-6287-4ba3-9467-93f84ae883ee-kube-api-access-tk9sv\") pod \"premium-w3-061fab-kserve-7bb7d67d85-92kbb\" (UID: \"c9628178-6287-4ba3-9467-93f84ae883ee\") " pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:47:35.104486 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:35.104387    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-model-cache\") pod \"premium-w3-061fab-kserve-7bb7d67d85-92kbb\" (UID: \"c9628178-6287-4ba3-9467-93f84ae883ee\") " pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:47:35.104486 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:35.104429    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9628178-6287-4ba3-9467-93f84ae883ee-tls-certs\") pod \"premium-w3-061fab-kserve-7bb7d67d85-92kbb\" (UID: \"c9628178-6287-4ba3-9467-93f84ae883ee\") " pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:47:35.104754 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:35.104733    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-tmp-dir\") pod \"premium-w3-061fab-kserve-7bb7d67d85-92kbb\" (UID: \"c9628178-6287-4ba3-9467-93f84ae883ee\") " pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:47:35.104806 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:35.104783    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-home\") pod \"premium-w3-061fab-kserve-7bb7d67d85-92kbb\" (UID: \"c9628178-6287-4ba3-9467-93f84ae883ee\") " pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:47:35.104845 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:35.104831    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-model-cache\") pod \"premium-w3-061fab-kserve-7bb7d67d85-92kbb\" (UID: \"c9628178-6287-4ba3-9467-93f84ae883ee\") " pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:47:35.105333 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:35.105311    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-061fab\"/\"default-dockercfg-qlw9w\""
Nov 02 01:47:35.105492 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:35.105414    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-061fab\"/\"premium-w3-061fab-kserve-self-signed-certs\""
Nov 02 01:47:35.105680 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:35.105663    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"]
Nov 02 01:47:35.107233 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:35.107202    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-dshm\") pod \"premium-w3-061fab-kserve-7bb7d67d85-92kbb\" (UID: \"c9628178-6287-4ba3-9467-93f84ae883ee\") " pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:47:35.117181 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:35.117163    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9628178-6287-4ba3-9467-93f84ae883ee-tls-certs\") pod \"premium-w3-061fab-kserve-7bb7d67d85-92kbb\" (UID: \"c9628178-6287-4ba3-9467-93f84ae883ee\") " pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:47:35.168713 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:35.168679    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk9sv\" (UniqueName: \"kubernetes.io/projected/c9628178-6287-4ba3-9467-93f84ae883ee-kube-api-access-tk9sv\") pod \"premium-w3-061fab-kserve-7bb7d67d85-92kbb\" (UID: \"c9628178-6287-4ba3-9467-93f84ae883ee\") " pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:47:35.312346 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:35.312277    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:47:35.714467 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:35.714441    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"]
Nov 02 01:47:35.717026 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:47:35.716991    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9628178_6287_4ba3_9467_93f84ae883ee.slice/crio-afb681e048511ef52f61a035dfee90c86ce83e8439470a564f11cc735cbcf262 WatchSource:0}: Error finding container afb681e048511ef52f61a035dfee90c86ce83e8439470a564f11cc735cbcf262: Status 404 returned error can't find the container with id afb681e048511ef52f61a035dfee90c86ce83e8439470a564f11cc735cbcf262
Nov 02 01:47:36.339903 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:36.339863    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb" event={"ID":"c9628178-6287-4ba3-9467-93f84ae883ee","Type":"ContainerStarted","Data":"aa26d60988af4c46d1e6754d72c47c81c28f6a19719c661dce559007c3422cd9"}
Nov 02 01:47:36.340294 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:36.339912    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb" event={"ID":"c9628178-6287-4ba3-9467-93f84ae883ee","Type":"ContainerStarted","Data":"afb681e048511ef52f61a035dfee90c86ce83e8439470a564f11cc735cbcf262"}
Nov 02 01:47:36.457839 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:36.457788    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb" podStartSLOduration=2.457770168 podStartE2EDuration="2.457770168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:47:36.456877133 +0000 UTC m=+903.897913630" watchObservedRunningTime="2026-11-02 01:47:36.457770168 +0000 UTC m=+903.898806663"
Nov 02 01:47:37.312401 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:37.312370    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:47:37.879846 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:37.879819    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-796832db-r8xr4"]
Nov 02 01:47:37.883838 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:37.883823    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796832db-r8xr4"
Nov 02 01:47:37.935056 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:37.935024    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635321b3-6294-40e6-bca9-d68130ba9751-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-796832db-r8xr4\" (UID: \"635321b3-6294-40e6-bca9-d68130ba9751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796832db-r8xr4"
Nov 02 01:47:37.935208 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:37.935125    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxh84\" (UniqueName: \"kubernetes.io/projected/635321b3-6294-40e6-bca9-d68130ba9751-kube-api-access-cxh84\") pod \"maas-api-revoke-sub-796832db-r8xr4\" (UID: \"635321b3-6294-40e6-bca9-d68130ba9751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796832db-r8xr4"
Nov 02 01:47:37.935208 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:37.935160    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635321b3-6294-40e6-bca9-d68130ba9751-maas-api-xks-ca\") pod \"maas-api-revoke-sub-796832db-r8xr4\" (UID: \"635321b3-6294-40e6-bca9-d68130ba9751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796832db-r8xr4"
Nov 02 01:47:37.941176 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:37.941151    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-796832db-r8xr4"]
Nov 02 01:47:37.987798 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:37.987768    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:47:37.987959 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:37.987857    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:47:38.036101 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:38.036067    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635321b3-6294-40e6-bca9-d68130ba9751-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-796832db-r8xr4\" (UID: \"635321b3-6294-40e6-bca9-d68130ba9751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796832db-r8xr4"
Nov 02 01:47:38.036334 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:38.036127    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxh84\" (UniqueName: \"kubernetes.io/projected/635321b3-6294-40e6-bca9-d68130ba9751-kube-api-access-cxh84\") pod \"maas-api-revoke-sub-796832db-r8xr4\" (UID: \"635321b3-6294-40e6-bca9-d68130ba9751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796832db-r8xr4"
Nov 02 01:47:38.036334 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:38.036261    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635321b3-6294-40e6-bca9-d68130ba9751-maas-api-xks-ca\") pod \"maas-api-revoke-sub-796832db-r8xr4\" (UID: \"635321b3-6294-40e6-bca9-d68130ba9751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796832db-r8xr4"
Nov 02 01:47:38.036813 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:38.036790    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635321b3-6294-40e6-bca9-d68130ba9751-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-796832db-r8xr4\" (UID: \"635321b3-6294-40e6-bca9-d68130ba9751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796832db-r8xr4"
Nov 02 01:47:38.038886 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:38.038862    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635321b3-6294-40e6-bca9-d68130ba9751-maas-api-xks-ca\") pod \"maas-api-revoke-sub-796832db-r8xr4\" (UID: \"635321b3-6294-40e6-bca9-d68130ba9751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796832db-r8xr4"
Nov 02 01:47:38.096836 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:38.096811    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxh84\" (UniqueName: \"kubernetes.io/projected/635321b3-6294-40e6-bca9-d68130ba9751-kube-api-access-cxh84\") pod \"maas-api-revoke-sub-796832db-r8xr4\" (UID: \"635321b3-6294-40e6-bca9-d68130ba9751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-796832db-r8xr4"
Nov 02 01:47:38.197550 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:38.197478    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796832db-r8xr4"
Nov 02 01:47:38.573893 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:38.573871    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-796832db-r8xr4"]
Nov 02 01:47:38.576359 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:47:38.576319    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635321b3_6294_40e6_bca9_d68130ba9751.slice/crio-bcbcff6cf520e1faac8df3b75a194407f04adb29f10411ba6d56b75127efdaeb WatchSource:0}: Error finding container bcbcff6cf520e1faac8df3b75a194407f04adb29f10411ba6d56b75127efdaeb: Status 404 returned error can't find the container with id bcbcff6cf520e1faac8df3b75a194407f04adb29f10411ba6d56b75127efdaeb
Nov 02 01:47:39.360465 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:39.360428    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796832db-r8xr4" event={"ID":"635321b3-6294-40e6-bca9-d68130ba9751","Type":"ContainerStarted","Data":"37ee72a67ec7d6cf5e012e41acbfa3a01f609088720740dd110656ce964517c2"}
Nov 02 01:47:39.360880 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:39.360483    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796832db-r8xr4" event={"ID":"635321b3-6294-40e6-bca9-d68130ba9751","Type":"ContainerStarted","Data":"bcbcff6cf520e1faac8df3b75a194407f04adb29f10411ba6d56b75127efdaeb"}
Nov 02 01:47:39.476585 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:39.476533    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796832db-r8xr4" podStartSLOduration=2.4765193 podStartE2EDuration="2.4765193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:47:39.47609085 +0000 UTC m=+906.917127350" watchObservedRunningTime="2026-11-02 01:47:39.4765193 +0000 UTC m=+906.917555797"
Nov 02 01:47:40.367471 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:40.367392    5875 generic.go:356] "Generic (PLEG): container finished" podID="635321b3-6294-40e6-bca9-d68130ba9751" containerID="37ee72a67ec7d6cf5e012e41acbfa3a01f609088720740dd110656ce964517c2" exitCode=0
Nov 02 01:47:40.367844 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:40.367484    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796832db-r8xr4" event={"ID":"635321b3-6294-40e6-bca9-d68130ba9751","Type":"ContainerDied","Data":"37ee72a67ec7d6cf5e012e41acbfa3a01f609088720740dd110656ce964517c2"}
Nov 02 01:47:41.319030 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:47:41.318992    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:47:41.515086 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:41.515061    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796832db-r8xr4"
Nov 02 01:47:41.572721 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:41.572658    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/635321b3-6294-40e6-bca9-d68130ba9751-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635321b3-6294-40e6-bca9-d68130ba9751-maas-api-openshift-ca\") pod \"635321b3-6294-40e6-bca9-d68130ba9751\" (UID: \"635321b3-6294-40e6-bca9-d68130ba9751\") "
Nov 02 01:47:41.572855 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:41.572723    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/635321b3-6294-40e6-bca9-d68130ba9751-kube-api-access-cxh84\" (UniqueName: \"kubernetes.io/projected/635321b3-6294-40e6-bca9-d68130ba9751-kube-api-access-cxh84\") pod \"635321b3-6294-40e6-bca9-d68130ba9751\" (UID: \"635321b3-6294-40e6-bca9-d68130ba9751\") "
Nov 02 01:47:41.572855 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:41.572778    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/635321b3-6294-40e6-bca9-d68130ba9751-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635321b3-6294-40e6-bca9-d68130ba9751-maas-api-xks-ca\") pod \"635321b3-6294-40e6-bca9-d68130ba9751\" (UID: \"635321b3-6294-40e6-bca9-d68130ba9751\") "
Nov 02 01:47:41.572995 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:41.572964    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/635321b3-6294-40e6-bca9-d68130ba9751-maas-api-openshift-ca" pod "635321b3-6294-40e6-bca9-d68130ba9751" (UID: "635321b3-6294-40e6-bca9-d68130ba9751"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:47:41.573122 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:41.573106    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/635321b3-6294-40e6-bca9-d68130ba9751-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:41.574820 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:41.574800    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635321b3-6294-40e6-bca9-d68130ba9751-kube-api-access-cxh84" pod "635321b3-6294-40e6-bca9-d68130ba9751" (UID: "635321b3-6294-40e6-bca9-d68130ba9751"). InnerVolumeSpecName "kube-api-access-cxh84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:47:41.574919 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:41.574904    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/635321b3-6294-40e6-bca9-d68130ba9751-maas-api-xks-ca" pod "635321b3-6294-40e6-bca9-d68130ba9751" (UID: "635321b3-6294-40e6-bca9-d68130ba9751"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:47:41.673854 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:41.673823    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxh84\" (UniqueName: \"kubernetes.io/projected/635321b3-6294-40e6-bca9-d68130ba9751-kube-api-access-cxh84\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:41.673854 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:41.673849    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/635321b3-6294-40e6-bca9-d68130ba9751-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:42.380547 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:42.380507    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796832db-r8xr4" event={"ID":"635321b3-6294-40e6-bca9-d68130ba9751","Type":"ContainerDied","Data":"bcbcff6cf520e1faac8df3b75a194407f04adb29f10411ba6d56b75127efdaeb"}
Nov 02 01:47:42.380547 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:42.380536    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-796832db-r8xr4"
Nov 02 01:47:42.380547 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:42.380554    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcbcff6cf520e1faac8df3b75a194407f04adb29f10411ba6d56b75127efdaeb"
Nov 02 01:47:45.312982 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:45.312949    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:47:45.330536 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:45.330503    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:47:45.408097 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:45.408069    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:47:53.386824 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:53.386745    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fa78e83-8zxdb"]
Nov 02 01:47:53.387304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:53.387290    5875 cpu_manager.go:414] "containerMap: removing container" podUID="635321b3-6294-40e6-bca9-d68130ba9751" containerName="revoke-keys"
Nov 02 01:47:53.387363 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:53.387305    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="635321b3-6294-40e6-bca9-d68130ba9751" containerName="revoke-keys"
Nov 02 01:47:53.387410 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:53.387401    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="635321b3-6294-40e6-bca9-d68130ba9751" containerName="revoke-keys"
Nov 02 01:47:53.390689 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:53.390675    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa78e83-8zxdb"
Nov 02 01:47:53.444641 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:53.444611    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:47:53.494380 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:53.494349    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:47:53.496270 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:53.496251    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fa78e83-8zxdb"]
Nov 02 01:47:53.507823 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:53.507801    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec7b2174-8ada-4eb8-b8e0-62b80ee430ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fa78e83-8zxdb\" (UID: \"ec7b2174-8ada-4eb8-b8e0-62b80ee430ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa78e83-8zxdb"
Nov 02 01:47:53.507915 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:53.507839    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rtng\" (UniqueName: \"kubernetes.io/projected/ec7b2174-8ada-4eb8-b8e0-62b80ee430ab-kube-api-access-5rtng\") pod \"maas-api-revoke-sub-2fa78e83-8zxdb\" (UID: \"ec7b2174-8ada-4eb8-b8e0-62b80ee430ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa78e83-8zxdb"
Nov 02 01:47:53.507964 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:53.507921    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec7b2174-8ada-4eb8-b8e0-62b80ee430ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fa78e83-8zxdb\" (UID: \"ec7b2174-8ada-4eb8-b8e0-62b80ee430ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa78e83-8zxdb"
Nov 02 01:47:53.608588 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:53.608561    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec7b2174-8ada-4eb8-b8e0-62b80ee430ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fa78e83-8zxdb\" (UID: \"ec7b2174-8ada-4eb8-b8e0-62b80ee430ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa78e83-8zxdb"
Nov 02 01:47:53.608766 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:53.608604    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rtng\" (UniqueName: \"kubernetes.io/projected/ec7b2174-8ada-4eb8-b8e0-62b80ee430ab-kube-api-access-5rtng\") pod \"maas-api-revoke-sub-2fa78e83-8zxdb\" (UID: \"ec7b2174-8ada-4eb8-b8e0-62b80ee430ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa78e83-8zxdb"
Nov 02 01:47:53.608766 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:53.608654    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec7b2174-8ada-4eb8-b8e0-62b80ee430ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fa78e83-8zxdb\" (UID: \"ec7b2174-8ada-4eb8-b8e0-62b80ee430ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa78e83-8zxdb"
Nov 02 01:47:53.609350 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:53.609330    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec7b2174-8ada-4eb8-b8e0-62b80ee430ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fa78e83-8zxdb\" (UID: \"ec7b2174-8ada-4eb8-b8e0-62b80ee430ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa78e83-8zxdb"
Nov 02 01:47:53.611072 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:53.611052    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec7b2174-8ada-4eb8-b8e0-62b80ee430ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fa78e83-8zxdb\" (UID: \"ec7b2174-8ada-4eb8-b8e0-62b80ee430ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa78e83-8zxdb"
Nov 02 01:47:53.669835 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:53.669775    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rtng\" (UniqueName: \"kubernetes.io/projected/ec7b2174-8ada-4eb8-b8e0-62b80ee430ab-kube-api-access-5rtng\") pod \"maas-api-revoke-sub-2fa78e83-8zxdb\" (UID: \"ec7b2174-8ada-4eb8-b8e0-62b80ee430ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa78e83-8zxdb"
Nov 02 01:47:53.703767 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:53.703741    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa78e83-8zxdb"
Nov 02 01:47:54.076103 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:54.076079    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fa78e83-8zxdb"]
Nov 02 01:47:54.078865 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:47:54.078828    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec7b2174_8ada_4eb8_b8e0_62b80ee430ab.slice/crio-f2ce29e884235072651f4aa1cfde52bfbeb0395723e49dda7e79120b466e0ce2 WatchSource:0}: Error finding container f2ce29e884235072651f4aa1cfde52bfbeb0395723e49dda7e79120b466e0ce2: Status 404 returned error can't find the container with id f2ce29e884235072651f4aa1cfde52bfbeb0395723e49dda7e79120b466e0ce2
Nov 02 01:47:54.455207 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:54.455177    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa78e83-8zxdb" event={"ID":"ec7b2174-8ada-4eb8-b8e0-62b80ee430ab","Type":"ContainerStarted","Data":"6793b50bbf0cdbf7e80b3e3f1f6106b81f93a4d06a6284c50e48ec8b2df9d1fa"}
Nov 02 01:47:54.455586 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:54.455266    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa78e83-8zxdb" event={"ID":"ec7b2174-8ada-4eb8-b8e0-62b80ee430ab","Type":"ContainerStarted","Data":"f2ce29e884235072651f4aa1cfde52bfbeb0395723e49dda7e79120b466e0ce2"}
Nov 02 01:47:54.605469 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:54.605418    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa78e83-8zxdb" podStartSLOduration=1.605404373 podStartE2EDuration="1.605404373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:47:54.603739566 +0000 UTC m=+922.044776054" watchObservedRunningTime="2026-11-02 01:47:54.605404373 +0000 UTC m=+922.046440908"
Nov 02 01:47:55.462966 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:55.462936    5875 generic.go:356] "Generic (PLEG): container finished" podID="ec7b2174-8ada-4eb8-b8e0-62b80ee430ab" containerID="6793b50bbf0cdbf7e80b3e3f1f6106b81f93a4d06a6284c50e48ec8b2df9d1fa" exitCode=0
Nov 02 01:47:55.463354 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:55.463023    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa78e83-8zxdb" event={"ID":"ec7b2174-8ada-4eb8-b8e0-62b80ee430ab","Type":"ContainerDied","Data":"6793b50bbf0cdbf7e80b3e3f1f6106b81f93a4d06a6284c50e48ec8b2df9d1fa"}
Nov 02 01:47:56.616164 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:56.616141    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa78e83-8zxdb"
Nov 02 01:47:56.737634 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:56.737555    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec7b2174-8ada-4eb8-b8e0-62b80ee430ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec7b2174-8ada-4eb8-b8e0-62b80ee430ab-maas-api-xks-ca\") pod \"ec7b2174-8ada-4eb8-b8e0-62b80ee430ab\" (UID: \"ec7b2174-8ada-4eb8-b8e0-62b80ee430ab\") "
Nov 02 01:47:56.737796 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:56.737656    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec7b2174-8ada-4eb8-b8e0-62b80ee430ab-kube-api-access-5rtng\" (UniqueName: \"kubernetes.io/projected/ec7b2174-8ada-4eb8-b8e0-62b80ee430ab-kube-api-access-5rtng\") pod \"ec7b2174-8ada-4eb8-b8e0-62b80ee430ab\" (UID: \"ec7b2174-8ada-4eb8-b8e0-62b80ee430ab\") "
Nov 02 01:47:56.737796 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:56.737689    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec7b2174-8ada-4eb8-b8e0-62b80ee430ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec7b2174-8ada-4eb8-b8e0-62b80ee430ab-maas-api-openshift-ca\") pod \"ec7b2174-8ada-4eb8-b8e0-62b80ee430ab\" (UID: \"ec7b2174-8ada-4eb8-b8e0-62b80ee430ab\") "
Nov 02 01:47:56.738442 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:56.738389    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec7b2174-8ada-4eb8-b8e0-62b80ee430ab-maas-api-openshift-ca" pod "ec7b2174-8ada-4eb8-b8e0-62b80ee430ab" (UID: "ec7b2174-8ada-4eb8-b8e0-62b80ee430ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:47:56.740673 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:56.740643    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec7b2174-8ada-4eb8-b8e0-62b80ee430ab-maas-api-xks-ca" pod "ec7b2174-8ada-4eb8-b8e0-62b80ee430ab" (UID: "ec7b2174-8ada-4eb8-b8e0-62b80ee430ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:47:56.740673 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:56.740661    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec7b2174-8ada-4eb8-b8e0-62b80ee430ab-kube-api-access-5rtng" pod "ec7b2174-8ada-4eb8-b8e0-62b80ee430ab" (UID: "ec7b2174-8ada-4eb8-b8e0-62b80ee430ab"). InnerVolumeSpecName "kube-api-access-5rtng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:47:56.838982 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:56.838952    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec7b2174-8ada-4eb8-b8e0-62b80ee430ab-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:56.838982 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:56.838977    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rtng\" (UniqueName: \"kubernetes.io/projected/ec7b2174-8ada-4eb8-b8e0-62b80ee430ab-kube-api-access-5rtng\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:56.838982 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:56.838988    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec7b2174-8ada-4eb8-b8e0-62b80ee430ab-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:47:57.476729 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:57.476703    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa78e83-8zxdb"
Nov 02 01:47:57.476729 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:57.476707    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fa78e83-8zxdb" event={"ID":"ec7b2174-8ada-4eb8-b8e0-62b80ee430ab","Type":"ContainerDied","Data":"f2ce29e884235072651f4aa1cfde52bfbeb0395723e49dda7e79120b466e0ce2"}
Nov 02 01:47:57.476987 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:57.476756    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2ce29e884235072651f4aa1cfde52bfbeb0395723e49dda7e79120b466e0ce2"
Nov 02 01:47:59.386665 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:47:59.386621    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:48:04.631825 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:04.631788    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66d2ec55-rrr6v"]
Nov 02 01:48:04.632324 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:04.632311    5875 cpu_manager.go:414] "containerMap: removing container" podUID="ec7b2174-8ada-4eb8-b8e0-62b80ee430ab" containerName="revoke-keys"
Nov 02 01:48:04.632380 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:04.632326    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec7b2174-8ada-4eb8-b8e0-62b80ee430ab" containerName="revoke-keys"
Nov 02 01:48:04.632442 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:04.632433    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec7b2174-8ada-4eb8-b8e0-62b80ee430ab" containerName="revoke-keys"
Nov 02 01:48:04.635737 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:04.635719    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d2ec55-rrr6v"
Nov 02 01:48:04.722950 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:04.722901    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/661e73e9-8fde-4f05-9fd8-e5a182b5fdc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66d2ec55-rrr6v\" (UID: \"661e73e9-8fde-4f05-9fd8-e5a182b5fdc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d2ec55-rrr6v"
Nov 02 01:48:04.723137 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:04.722994    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfmfh\" (UniqueName: \"kubernetes.io/projected/661e73e9-8fde-4f05-9fd8-e5a182b5fdc6-kube-api-access-vfmfh\") pod \"maas-api-revoke-sub-66d2ec55-rrr6v\" (UID: \"661e73e9-8fde-4f05-9fd8-e5a182b5fdc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d2ec55-rrr6v"
Nov 02 01:48:04.723137 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:04.723068    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/661e73e9-8fde-4f05-9fd8-e5a182b5fdc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66d2ec55-rrr6v\" (UID: \"661e73e9-8fde-4f05-9fd8-e5a182b5fdc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d2ec55-rrr6v"
Nov 02 01:48:04.739278 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:04.739252    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:48:04.739398 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:04.739340    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:48:04.741660 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:04.741635    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66d2ec55-rrr6v"]
Nov 02 01:48:04.824055 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:04.824034    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfmfh\" (UniqueName: \"kubernetes.io/projected/661e73e9-8fde-4f05-9fd8-e5a182b5fdc6-kube-api-access-vfmfh\") pod \"maas-api-revoke-sub-66d2ec55-rrr6v\" (UID: \"661e73e9-8fde-4f05-9fd8-e5a182b5fdc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d2ec55-rrr6v"
Nov 02 01:48:04.824180 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:04.824080    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/661e73e9-8fde-4f05-9fd8-e5a182b5fdc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66d2ec55-rrr6v\" (UID: \"661e73e9-8fde-4f05-9fd8-e5a182b5fdc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d2ec55-rrr6v"
Nov 02 01:48:04.824180 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:04.824133    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/661e73e9-8fde-4f05-9fd8-e5a182b5fdc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66d2ec55-rrr6v\" (UID: \"661e73e9-8fde-4f05-9fd8-e5a182b5fdc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d2ec55-rrr6v"
Nov 02 01:48:04.824786 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:04.824766    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/661e73e9-8fde-4f05-9fd8-e5a182b5fdc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66d2ec55-rrr6v\" (UID: \"661e73e9-8fde-4f05-9fd8-e5a182b5fdc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d2ec55-rrr6v"
Nov 02 01:48:04.826531 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:04.826515    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/661e73e9-8fde-4f05-9fd8-e5a182b5fdc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66d2ec55-rrr6v\" (UID: \"661e73e9-8fde-4f05-9fd8-e5a182b5fdc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d2ec55-rrr6v"
Nov 02 01:48:04.884831 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:04.884770    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfmfh\" (UniqueName: \"kubernetes.io/projected/661e73e9-8fde-4f05-9fd8-e5a182b5fdc6-kube-api-access-vfmfh\") pod \"maas-api-revoke-sub-66d2ec55-rrr6v\" (UID: \"661e73e9-8fde-4f05-9fd8-e5a182b5fdc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d2ec55-rrr6v"
Nov 02 01:48:04.948503 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:04.948479    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d2ec55-rrr6v"
Nov 02 01:48:05.322272 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:05.322239    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66d2ec55-rrr6v"]
Nov 02 01:48:05.328298 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:48:05.328263    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661e73e9_8fde_4f05_9fd8_e5a182b5fdc6.slice/crio-1571f8c6ca33a08c63bf04691b979a442b605f56739e05fe211aece8dc795927 WatchSource:0}: Error finding container 1571f8c6ca33a08c63bf04691b979a442b605f56739e05fe211aece8dc795927: Status 404 returned error can't find the container with id 1571f8c6ca33a08c63bf04691b979a442b605f56739e05fe211aece8dc795927
Nov 02 01:48:05.522752 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:05.522724    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d2ec55-rrr6v" event={"ID":"661e73e9-8fde-4f05-9fd8-e5a182b5fdc6","Type":"ContainerStarted","Data":"1571f8c6ca33a08c63bf04691b979a442b605f56739e05fe211aece8dc795927"}
Nov 02 01:48:06.529868 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:06.529825    5875 generic.go:356] "Generic (PLEG): container finished" podID="661e73e9-8fde-4f05-9fd8-e5a182b5fdc6" containerID="ddb6604510e3dce680ff827c6f49ad1cde01147e272fe780f8541dc7523df05f" exitCode=0
Nov 02 01:48:06.530339 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:06.529910    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d2ec55-rrr6v" event={"ID":"661e73e9-8fde-4f05-9fd8-e5a182b5fdc6","Type":"ContainerDied","Data":"ddb6604510e3dce680ff827c6f49ad1cde01147e272fe780f8541dc7523df05f"}
Nov 02 01:48:07.680453 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:07.680422    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d2ec55-rrr6v"
Nov 02 01:48:07.856671 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:07.856591    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/661e73e9-8fde-4f05-9fd8-e5a182b5fdc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/661e73e9-8fde-4f05-9fd8-e5a182b5fdc6-maas-api-xks-ca\") pod \"661e73e9-8fde-4f05-9fd8-e5a182b5fdc6\" (UID: \"661e73e9-8fde-4f05-9fd8-e5a182b5fdc6\") "
Nov 02 01:48:07.856847 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:07.856727    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/661e73e9-8fde-4f05-9fd8-e5a182b5fdc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/661e73e9-8fde-4f05-9fd8-e5a182b5fdc6-maas-api-openshift-ca\") pod \"661e73e9-8fde-4f05-9fd8-e5a182b5fdc6\" (UID: \"661e73e9-8fde-4f05-9fd8-e5a182b5fdc6\") "
Nov 02 01:48:07.856847 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:07.856800    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/661e73e9-8fde-4f05-9fd8-e5a182b5fdc6-kube-api-access-vfmfh\" (UniqueName: \"kubernetes.io/projected/661e73e9-8fde-4f05-9fd8-e5a182b5fdc6-kube-api-access-vfmfh\") pod \"661e73e9-8fde-4f05-9fd8-e5a182b5fdc6\" (UID: \"661e73e9-8fde-4f05-9fd8-e5a182b5fdc6\") "
Nov 02 01:48:07.857081 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:07.857059    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661e73e9-8fde-4f05-9fd8-e5a182b5fdc6-maas-api-openshift-ca" pod "661e73e9-8fde-4f05-9fd8-e5a182b5fdc6" (UID: "661e73e9-8fde-4f05-9fd8-e5a182b5fdc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:07.857182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:07.857165    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/661e73e9-8fde-4f05-9fd8-e5a182b5fdc6-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:07.858898 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:07.858878    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/661e73e9-8fde-4f05-9fd8-e5a182b5fdc6-maas-api-xks-ca" pod "661e73e9-8fde-4f05-9fd8-e5a182b5fdc6" (UID: "661e73e9-8fde-4f05-9fd8-e5a182b5fdc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:07.858980 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:07.858920    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661e73e9-8fde-4f05-9fd8-e5a182b5fdc6-kube-api-access-vfmfh" pod "661e73e9-8fde-4f05-9fd8-e5a182b5fdc6" (UID: "661e73e9-8fde-4f05-9fd8-e5a182b5fdc6"). InnerVolumeSpecName "kube-api-access-vfmfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:07.958314 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:07.958286    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/661e73e9-8fde-4f05-9fd8-e5a182b5fdc6-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:07.958314 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:07.958310    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfmfh\" (UniqueName: \"kubernetes.io/projected/661e73e9-8fde-4f05-9fd8-e5a182b5fdc6-kube-api-access-vfmfh\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:08.543515 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:08.543471    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d2ec55-rrr6v" event={"ID":"661e73e9-8fde-4f05-9fd8-e5a182b5fdc6","Type":"ContainerDied","Data":"1571f8c6ca33a08c63bf04691b979a442b605f56739e05fe211aece8dc795927"}
Nov 02 01:48:08.543515 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:08.543490    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d2ec55-rrr6v"
Nov 02 01:48:08.543515 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:08.543521    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1571f8c6ca33a08c63bf04691b979a442b605f56739e05fe211aece8dc795927"
Nov 02 01:48:09.090441 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:09.090404    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b529ea0-6svnk"]
Nov 02 01:48:09.091158 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:09.091137    5875 cpu_manager.go:414] "containerMap: removing container" podUID="661e73e9-8fde-4f05-9fd8-e5a182b5fdc6" containerName="revoke-keys"
Nov 02 01:48:09.091277 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:09.091159    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="661e73e9-8fde-4f05-9fd8-e5a182b5fdc6" containerName="revoke-keys"
Nov 02 01:48:09.091380 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:09.091365    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="661e73e9-8fde-4f05-9fd8-e5a182b5fdc6" containerName="revoke-keys"
Nov 02 01:48:09.094511 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:09.094486    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529ea0-6svnk"
Nov 02 01:48:09.149287 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:09.149093    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:48:09.198337 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:09.198309    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:48:09.200925 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:09.200906    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b529ea0-6svnk"]
Nov 02 01:48:09.278762 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:09.278723    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr8t9\" (UniqueName: \"kubernetes.io/projected/2b2feeac-7626-4081-a2e7-16b141a5b146-kube-api-access-pr8t9\") pod \"maas-api-revoke-sub-7b529ea0-6svnk\" (UID: \"2b2feeac-7626-4081-a2e7-16b141a5b146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529ea0-6svnk"
Nov 02 01:48:09.279303 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:09.279277    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2feeac-7626-4081-a2e7-16b141a5b146-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b529ea0-6svnk\" (UID: \"2b2feeac-7626-4081-a2e7-16b141a5b146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529ea0-6svnk"
Nov 02 01:48:09.279483 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:09.279467    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2feeac-7626-4081-a2e7-16b141a5b146-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b529ea0-6svnk\" (UID: \"2b2feeac-7626-4081-a2e7-16b141a5b146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529ea0-6svnk"
Nov 02 01:48:09.380570 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:09.380497    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2feeac-7626-4081-a2e7-16b141a5b146-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b529ea0-6svnk\" (UID: \"2b2feeac-7626-4081-a2e7-16b141a5b146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529ea0-6svnk"
Nov 02 01:48:09.380570 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:09.380526    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2feeac-7626-4081-a2e7-16b141a5b146-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b529ea0-6svnk\" (UID: \"2b2feeac-7626-4081-a2e7-16b141a5b146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529ea0-6svnk"
Nov 02 01:48:09.380753 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:09.380615    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr8t9\" (UniqueName: \"kubernetes.io/projected/2b2feeac-7626-4081-a2e7-16b141a5b146-kube-api-access-pr8t9\") pod \"maas-api-revoke-sub-7b529ea0-6svnk\" (UID: \"2b2feeac-7626-4081-a2e7-16b141a5b146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529ea0-6svnk"
Nov 02 01:48:09.381154 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:09.381129    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2feeac-7626-4081-a2e7-16b141a5b146-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b529ea0-6svnk\" (UID: \"2b2feeac-7626-4081-a2e7-16b141a5b146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529ea0-6svnk"
Nov 02 01:48:09.383046 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:09.383028    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2feeac-7626-4081-a2e7-16b141a5b146-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b529ea0-6svnk\" (UID: \"2b2feeac-7626-4081-a2e7-16b141a5b146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529ea0-6svnk"
Nov 02 01:48:09.440628 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:09.440605    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr8t9\" (UniqueName: \"kubernetes.io/projected/2b2feeac-7626-4081-a2e7-16b141a5b146-kube-api-access-pr8t9\") pod \"maas-api-revoke-sub-7b529ea0-6svnk\" (UID: \"2b2feeac-7626-4081-a2e7-16b141a5b146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529ea0-6svnk"
Nov 02 01:48:09.708899 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:09.708873    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529ea0-6svnk"
Nov 02 01:48:10.084081 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:10.084057    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b529ea0-6svnk"]
Nov 02 01:48:10.085919 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:48:10.085891    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2feeac_7626_4081_a2e7_16b141a5b146.slice/crio-a3fa92b535a0139f8ffa0a5b6a3a433ba8babb71aab54e5a1bece22b5c45ce95 WatchSource:0}: Error finding container a3fa92b535a0139f8ffa0a5b6a3a433ba8babb71aab54e5a1bece22b5c45ce95: Status 404 returned error can't find the container with id a3fa92b535a0139f8ffa0a5b6a3a433ba8babb71aab54e5a1bece22b5c45ce95
Nov 02 01:48:10.561105 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:10.561066    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529ea0-6svnk" event={"ID":"2b2feeac-7626-4081-a2e7-16b141a5b146","Type":"ContainerStarted","Data":"97dcab673d795af310f792476a95272af9c27433e7076e5d5f4b96a2dcae2a16"}
Nov 02 01:48:10.561543 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:10.561159    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529ea0-6svnk" event={"ID":"2b2feeac-7626-4081-a2e7-16b141a5b146","Type":"ContainerStarted","Data":"a3fa92b535a0139f8ffa0a5b6a3a433ba8babb71aab54e5a1bece22b5c45ce95"}
Nov 02 01:48:10.726364 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:10.726312    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529ea0-6svnk" podStartSLOduration=1.726297228 podStartE2EDuration="1.726297228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:10.72491936 +0000 UTC m=+938.165955858" watchObservedRunningTime="2026-11-02 01:48:10.726297228 +0000 UTC m=+938.167333724"
Nov 02 01:48:11.318016 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:48:11.317936    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:48:11.568394 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:11.568304    5875 generic.go:356] "Generic (PLEG): container finished" podID="2b2feeac-7626-4081-a2e7-16b141a5b146" containerID="97dcab673d795af310f792476a95272af9c27433e7076e5d5f4b96a2dcae2a16" exitCode=0
Nov 02 01:48:11.568394 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:11.568372    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529ea0-6svnk" event={"ID":"2b2feeac-7626-4081-a2e7-16b141a5b146","Type":"ContainerDied","Data":"97dcab673d795af310f792476a95272af9c27433e7076e5d5f4b96a2dcae2a16"}
Nov 02 01:48:12.708274 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:12.708246    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529ea0-6svnk"
Nov 02 01:48:12.815067 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:12.815033    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b2feeac-7626-4081-a2e7-16b141a5b146-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2feeac-7626-4081-a2e7-16b141a5b146-maas-api-openshift-ca\") pod \"2b2feeac-7626-4081-a2e7-16b141a5b146\" (UID: \"2b2feeac-7626-4081-a2e7-16b141a5b146\") "
Nov 02 01:48:12.815293 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:12.815133    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b2feeac-7626-4081-a2e7-16b141a5b146-kube-api-access-pr8t9\" (UniqueName: \"kubernetes.io/projected/2b2feeac-7626-4081-a2e7-16b141a5b146-kube-api-access-pr8t9\") pod \"2b2feeac-7626-4081-a2e7-16b141a5b146\" (UID: \"2b2feeac-7626-4081-a2e7-16b141a5b146\") "
Nov 02 01:48:12.815293 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:12.815283    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b2feeac-7626-4081-a2e7-16b141a5b146-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2feeac-7626-4081-a2e7-16b141a5b146-maas-api-xks-ca\") pod \"2b2feeac-7626-4081-a2e7-16b141a5b146\" (UID: \"2b2feeac-7626-4081-a2e7-16b141a5b146\") "
Nov 02 01:48:12.815490 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:12.815462    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b2feeac-7626-4081-a2e7-16b141a5b146-maas-api-openshift-ca" pod "2b2feeac-7626-4081-a2e7-16b141a5b146" (UID: "2b2feeac-7626-4081-a2e7-16b141a5b146"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:12.815808 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:12.815775    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2feeac-7626-4081-a2e7-16b141a5b146-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:12.817619 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:12.817592    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2feeac-7626-4081-a2e7-16b141a5b146-kube-api-access-pr8t9" pod "2b2feeac-7626-4081-a2e7-16b141a5b146" (UID: "2b2feeac-7626-4081-a2e7-16b141a5b146"). InnerVolumeSpecName "kube-api-access-pr8t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:12.817913 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:12.817890    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b2feeac-7626-4081-a2e7-16b141a5b146-maas-api-xks-ca" pod "2b2feeac-7626-4081-a2e7-16b141a5b146" (UID: "2b2feeac-7626-4081-a2e7-16b141a5b146"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:12.917245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:12.917193    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2feeac-7626-4081-a2e7-16b141a5b146-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:12.917245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:12.917242    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr8t9\" (UniqueName: \"kubernetes.io/projected/2b2feeac-7626-4081-a2e7-16b141a5b146-kube-api-access-pr8t9\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:13.581584 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:13.581555    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529ea0-6svnk"
Nov 02 01:48:13.581584 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:13.581574    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b529ea0-6svnk" event={"ID":"2b2feeac-7626-4081-a2e7-16b141a5b146","Type":"ContainerDied","Data":"a3fa92b535a0139f8ffa0a5b6a3a433ba8babb71aab54e5a1bece22b5c45ce95"}
Nov 02 01:48:13.581795 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:13.581615    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3fa92b535a0139f8ffa0a5b6a3a433ba8babb71aab54e5a1bece22b5c45ce95"
Nov 02 01:48:19.722792 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.722756    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"]
Nov 02 01:48:19.723306 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.723291    5875 cpu_manager.go:414] "containerMap: removing container" podUID="2b2feeac-7626-4081-a2e7-16b141a5b146" containerName="revoke-keys"
Nov 02 01:48:19.723359 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.723307    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b2feeac-7626-4081-a2e7-16b141a5b146" containerName="revoke-keys"
Nov 02 01:48:19.723411 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.723402    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b2feeac-7626-4081-a2e7-16b141a5b146" containerName="revoke-keys"
Nov 02 01:48:19.726745 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.726727    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.791422 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.791383    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-podinfo\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.791584 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.791508    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-envoy\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.791584 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.791562    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istiod-ca-cert\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.791715 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.791600    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-workload-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.791715 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.791653    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-credential-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.791832 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.791814    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-data\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.791906 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.791887    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxd2h\" (UniqueName: \"kubernetes.io/projected/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-kube-api-access-lxd2h\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.791985 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.791968    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-token\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.792030 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.792015    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-workload-certs\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.829426 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.828167    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9470a3-openshift-default-dockercfg-pfjf7\""
Nov 02 01:48:19.830875 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.830854    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"]
Nov 02 01:48:19.892774 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.892729    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-data\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.892936 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.892813    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxd2h\" (UniqueName: \"kubernetes.io/projected/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-kube-api-access-lxd2h\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.892936 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.892881    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-token\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.892936 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.892911    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-workload-certs\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.893085 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.892946    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-podinfo\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.893085 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.893056    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-envoy\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.893177 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.893108    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istiod-ca-cert\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.893177 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.893152    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-workload-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.893316 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.893178    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-credential-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.893316 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.893277    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-workload-certs\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.893551 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.893522    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-data\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.893635 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.893561    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-workload-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.893635 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.893580    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-credential-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.894170 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.894146    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istiod-ca-cert\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.895646 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.895624    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-envoy\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.895862 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.895843    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-podinfo\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.954780 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.954752    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-token\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:19.954910 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:19.954893    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxd2h\" (UniqueName: \"kubernetes.io/projected/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-kube-api-access-lxd2h\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:20.041179 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:20.041100    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:20.470058 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:20.470032    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"]
Nov 02 01:48:20.471505 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:48:20.471478    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5f896f_d35c_4cbe_87e2_f18aa35b5a37.slice/crio-97514ae56fba0b2933fcb454f1c953b3cb05d14ce67c1ef428168fd2a19db268 WatchSource:0}: Error finding container 97514ae56fba0b2933fcb454f1c953b3cb05d14ce67c1ef428168fd2a19db268: Status 404 returned error can't find the container with id 97514ae56fba0b2933fcb454f1c953b3cb05d14ce67c1ef428168fd2a19db268
Nov 02 01:48:20.473955 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:20.473925    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:48:20.474042 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:20.473986    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:48:20.474042 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:20.474013    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:48:20.625407 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:20.625372    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk" event={"ID":"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37","Type":"ContainerStarted","Data":"5fd885597ea0f87801233ad4c73a12efc8774735e406ba072a4100d700fc9106"}
Nov 02 01:48:20.625577 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:20.625431    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk" event={"ID":"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37","Type":"ContainerStarted","Data":"97514ae56fba0b2933fcb454f1c953b3cb05d14ce67c1ef428168fd2a19db268"}
Nov 02 01:48:20.806170 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:20.806057    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk" podStartSLOduration=1.806041609 podStartE2EDuration="1.806041609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:20.755569542 +0000 UTC m=+948.196606076" watchObservedRunningTime="2026-11-02 01:48:20.806041609 +0000 UTC m=+948.247078110"
Nov 02 01:48:21.042339 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:21.042212    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:21.043660 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:21.043635    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 01:48:21.043766 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:21.043684    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk" podUID="9e5f896f-d35c-4cbe-87e2-f18aa35b5a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 01:48:22.041737 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:22.041697    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 01:48:22.042144 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:22.041756    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk" podUID="9e5f896f-d35c-4cbe-87e2-f18aa35b5a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 01:48:23.041471 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:23.041439    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 01:48:23.041658 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:23.041491    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk" podUID="9e5f896f-d35c-4cbe-87e2-f18aa35b5a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 01:48:24.041723 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:24.041691    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 01:48:24.042103 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:24.041741    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk" podUID="9e5f896f-d35c-4cbe-87e2-f18aa35b5a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 01:48:25.042244 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:25.042179    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 01:48:25.042674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:25.042254    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk" podUID="9e5f896f-d35c-4cbe-87e2-f18aa35b5a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 01:48:25.057874 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:25.057847    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92e7c586-zx5vs"]
Nov 02 01:48:25.061862 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:25.061847    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e7c586-zx5vs"
Nov 02 01:48:25.126136 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:25.126103    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:48:25.131993 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:25.129125    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92e7c586-zx5vs"]
Nov 02 01:48:25.165153 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:25.165128    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:48:25.166643 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:25.166623    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd89e9b-403f-41e7-889b-615ebe4e5b24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92e7c586-zx5vs\" (UID: \"dcd89e9b-403f-41e7-889b-615ebe4e5b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e7c586-zx5vs"
Nov 02 01:48:25.166716 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:25.166660    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgx5z\" (UniqueName: \"kubernetes.io/projected/dcd89e9b-403f-41e7-889b-615ebe4e5b24-kube-api-access-xgx5z\") pod \"maas-api-revoke-sub-92e7c586-zx5vs\" (UID: \"dcd89e9b-403f-41e7-889b-615ebe4e5b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e7c586-zx5vs"
Nov 02 01:48:25.166831 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:25.166810    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd89e9b-403f-41e7-889b-615ebe4e5b24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92e7c586-zx5vs\" (UID: \"dcd89e9b-403f-41e7-889b-615ebe4e5b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e7c586-zx5vs"
Nov 02 01:48:25.267956 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:25.267921    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd89e9b-403f-41e7-889b-615ebe4e5b24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92e7c586-zx5vs\" (UID: \"dcd89e9b-403f-41e7-889b-615ebe4e5b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e7c586-zx5vs"
Nov 02 01:48:25.268168 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:25.268045    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd89e9b-403f-41e7-889b-615ebe4e5b24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92e7c586-zx5vs\" (UID: \"dcd89e9b-403f-41e7-889b-615ebe4e5b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e7c586-zx5vs"
Nov 02 01:48:25.268168 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:25.268070    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgx5z\" (UniqueName: \"kubernetes.io/projected/dcd89e9b-403f-41e7-889b-615ebe4e5b24-kube-api-access-xgx5z\") pod \"maas-api-revoke-sub-92e7c586-zx5vs\" (UID: \"dcd89e9b-403f-41e7-889b-615ebe4e5b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e7c586-zx5vs"
Nov 02 01:48:25.268744 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:25.268716    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd89e9b-403f-41e7-889b-615ebe4e5b24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92e7c586-zx5vs\" (UID: \"dcd89e9b-403f-41e7-889b-615ebe4e5b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e7c586-zx5vs"
Nov 02 01:48:25.270548 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:25.270527    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd89e9b-403f-41e7-889b-615ebe4e5b24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92e7c586-zx5vs\" (UID: \"dcd89e9b-403f-41e7-889b-615ebe4e5b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e7c586-zx5vs"
Nov 02 01:48:25.328988 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:25.328921    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgx5z\" (UniqueName: \"kubernetes.io/projected/dcd89e9b-403f-41e7-889b-615ebe4e5b24-kube-api-access-xgx5z\") pod \"maas-api-revoke-sub-92e7c586-zx5vs\" (UID: \"dcd89e9b-403f-41e7-889b-615ebe4e5b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e7c586-zx5vs"
Nov 02 01:48:25.375413 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:25.375365    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e7c586-zx5vs"
Nov 02 01:48:25.785892 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:25.785867    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92e7c586-zx5vs"]
Nov 02 01:48:25.787981 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:48:25.787955    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd89e9b_403f_41e7_889b_615ebe4e5b24.slice/crio-f808758abddc431b9790893dc46311a48fc21d35f2262757255894f6755963d1 WatchSource:0}: Error finding container f808758abddc431b9790893dc46311a48fc21d35f2262757255894f6755963d1: Status 404 returned error can't find the container with id f808758abddc431b9790893dc46311a48fc21d35f2262757255894f6755963d1
Nov 02 01:48:26.042269 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:26.042242    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 01:48:26.042600 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:26.042292    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk" podUID="9e5f896f-d35c-4cbe-87e2-f18aa35b5a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 01:48:26.662907 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:26.662869    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e7c586-zx5vs" event={"ID":"dcd89e9b-403f-41e7-889b-615ebe4e5b24","Type":"ContainerStarted","Data":"695b221e97ea7d7e9d61b412aa4d39052514b7afe95b1cd9999324276d7f7206"}
Nov 02 01:48:26.663069 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:26.662933    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e7c586-zx5vs" event={"ID":"dcd89e9b-403f-41e7-889b-615ebe4e5b24","Type":"ContainerStarted","Data":"f808758abddc431b9790893dc46311a48fc21d35f2262757255894f6755963d1"}
Nov 02 01:48:26.780474 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:26.780425    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e7c586-zx5vs" podStartSLOduration=1.78041 podStartE2EDuration="1.78041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:26.777086259 +0000 UTC m=+954.218122771" watchObservedRunningTime="2026-11-02 01:48:26.78041 +0000 UTC m=+954.221446496"
Nov 02 01:48:26.829531 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:26.829504    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"]
Nov 02 01:48:26.829754 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:26.829729    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589" podUID="407d3a7d-bc65-4626-8c49-3cb5091c32c4" containerName="istio-proxy" containerID="cri-o://9dbacd0fe02c38a82e961dc84897d35f964bafbe48d80c4df6ecc230190895a2" gracePeriod=30
Nov 02 01:48:27.124988 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:27.124954    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:27.125986 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:27.125959    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:27.126820 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:27.126797    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 01:48:27.670787 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:27.670752    5875 generic.go:356] "Generic (PLEG): container finished" podID="dcd89e9b-403f-41e7-889b-615ebe4e5b24" containerID="695b221e97ea7d7e9d61b412aa4d39052514b7afe95b1cd9999324276d7f7206" exitCode=0
Nov 02 01:48:27.670944 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:27.670833    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e7c586-zx5vs" event={"ID":"dcd89e9b-403f-41e7-889b-615ebe4e5b24","Type":"ContainerDied","Data":"695b221e97ea7d7e9d61b412aa4d39052514b7afe95b1cd9999324276d7f7206"}
Nov 02 01:48:28.827741 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:28.827717    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e7c586-zx5vs"
Nov 02 01:48:28.906231 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:28.906200    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcd89e9b-403f-41e7-889b-615ebe4e5b24-kube-api-access-xgx5z\" (UniqueName: \"kubernetes.io/projected/dcd89e9b-403f-41e7-889b-615ebe4e5b24-kube-api-access-xgx5z\") pod \"dcd89e9b-403f-41e7-889b-615ebe4e5b24\" (UID: \"dcd89e9b-403f-41e7-889b-615ebe4e5b24\") "
Nov 02 01:48:28.906406 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:28.906257    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcd89e9b-403f-41e7-889b-615ebe4e5b24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd89e9b-403f-41e7-889b-615ebe4e5b24-maas-api-openshift-ca\") pod \"dcd89e9b-403f-41e7-889b-615ebe4e5b24\" (UID: \"dcd89e9b-403f-41e7-889b-615ebe4e5b24\") "
Nov 02 01:48:28.906406 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:28.906299    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcd89e9b-403f-41e7-889b-615ebe4e5b24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd89e9b-403f-41e7-889b-615ebe4e5b24-maas-api-xks-ca\") pod \"dcd89e9b-403f-41e7-889b-615ebe4e5b24\" (UID: \"dcd89e9b-403f-41e7-889b-615ebe4e5b24\") "
Nov 02 01:48:28.906691 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:28.906663    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd89e9b-403f-41e7-889b-615ebe4e5b24-maas-api-openshift-ca" pod "dcd89e9b-403f-41e7-889b-615ebe4e5b24" (UID: "dcd89e9b-403f-41e7-889b-615ebe4e5b24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:28.908499 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:28.908479    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd89e9b-403f-41e7-889b-615ebe4e5b24-kube-api-access-xgx5z" pod "dcd89e9b-403f-41e7-889b-615ebe4e5b24" (UID: "dcd89e9b-403f-41e7-889b-615ebe4e5b24"). InnerVolumeSpecName "kube-api-access-xgx5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:28.908597 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:28.908511    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcd89e9b-403f-41e7-889b-615ebe4e5b24-maas-api-xks-ca" pod "dcd89e9b-403f-41e7-889b-615ebe4e5b24" (UID: "dcd89e9b-403f-41e7-889b-615ebe4e5b24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:48:29.007344 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:29.007315    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgx5z\" (UniqueName: \"kubernetes.io/projected/dcd89e9b-403f-41e7-889b-615ebe4e5b24-kube-api-access-xgx5z\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:29.007344 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:29.007340    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd89e9b-403f-41e7-889b-615ebe4e5b24-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:29.007344 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:29.007350    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd89e9b-403f-41e7-889b-615ebe4e5b24-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:29.683763 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:29.683734    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e7c586-zx5vs"
Nov 02 01:48:29.683960 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:29.683770    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e7c586-zx5vs" event={"ID":"dcd89e9b-403f-41e7-889b-615ebe4e5b24","Type":"ContainerDied","Data":"f808758abddc431b9790893dc46311a48fc21d35f2262757255894f6755963d1"}
Nov 02 01:48:29.683960 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:29.683815    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f808758abddc431b9790893dc46311a48fc21d35f2262757255894f6755963d1"
Nov 02 01:48:32.092470 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.092417    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:48:32.241455 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.241421    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-envoy\") pod \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") "
Nov 02 01:48:32.241628 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.241499    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-workload-socket\") pod \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") "
Nov 02 01:48:32.241628 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.241569    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-token\" (UniqueName: \"kubernetes.io/projected/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-token\") pod \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") "
Nov 02 01:48:32.241628 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.241600    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/407d3a7d-bc65-4626-8c49-3cb5091c32c4-kube-api-access-dvq5b\" (UniqueName: \"kubernetes.io/projected/407d3a7d-bc65-4626-8c49-3cb5091c32c4-kube-api-access-dvq5b\") pod \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") "
Nov 02 01:48:32.241808 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.241704    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-credential-socket\") pod \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") "
Nov 02 01:48:32.241808 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.241739    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-workload-certs\") pod \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") "
Nov 02 01:48:32.241808 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.241777    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istiod-ca-cert\") pod \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") "
Nov 02 01:48:32.241976 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.241812    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-workload-socket" pod "407d3a7d-bc65-4626-8c49-3cb5091c32c4" (UID: "407d3a7d-bc65-4626-8c49-3cb5091c32c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:32.241976 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.241872    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-podinfo\") pod \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") "
Nov 02 01:48:32.241976 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.241902    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-data\") pod \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\" (UID: \"407d3a7d-bc65-4626-8c49-3cb5091c32c4\") "
Nov 02 01:48:32.242140 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.242046    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-workload-certs" pod "407d3a7d-bc65-4626-8c49-3cb5091c32c4" (UID: "407d3a7d-bc65-4626-8c49-3cb5091c32c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:32.242288 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.242266    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istiod-ca-cert" pod "407d3a7d-bc65-4626-8c49-3cb5091c32c4" (UID: "407d3a7d-bc65-4626-8c49-3cb5091c32c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:32.242842 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.242362    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-data" pod "407d3a7d-bc65-4626-8c49-3cb5091c32c4" (UID: "407d3a7d-bc65-4626-8c49-3cb5091c32c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:32.242842 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.242368    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:32.242842 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.242457    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:32.243269 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.243212    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-credential-socket" pod "407d3a7d-bc65-4626-8c49-3cb5091c32c4" (UID: "407d3a7d-bc65-4626-8c49-3cb5091c32c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:32.245141 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.245113    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-token" pod "407d3a7d-bc65-4626-8c49-3cb5091c32c4" (UID: "407d3a7d-bc65-4626-8c49-3cb5091c32c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:32.245313 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.245214    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407d3a7d-bc65-4626-8c49-3cb5091c32c4-kube-api-access-dvq5b" pod "407d3a7d-bc65-4626-8c49-3cb5091c32c4" (UID: "407d3a7d-bc65-4626-8c49-3cb5091c32c4"). InnerVolumeSpecName "kube-api-access-dvq5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:32.245425 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.245379    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-envoy" pod "407d3a7d-bc65-4626-8c49-3cb5091c32c4" (UID: "407d3a7d-bc65-4626-8c49-3cb5091c32c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:32.245493 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.245442    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-podinfo" pod "407d3a7d-bc65-4626-8c49-3cb5091c32c4" (UID: "407d3a7d-bc65-4626-8c49-3cb5091c32c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:48:32.343935 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.343854    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:32.343935 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.343880    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:32.343935 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.343889    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:32.343935 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.343898    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:32.343935 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.343909    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:32.343935 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.343922    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/407d3a7d-bc65-4626-8c49-3cb5091c32c4-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:32.343935 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.343934    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvq5b\" (UniqueName: \"kubernetes.io/projected/407d3a7d-bc65-4626-8c49-3cb5091c32c4-kube-api-access-dvq5b\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:32.711999 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.711964    5875 generic.go:356] "Generic (PLEG): container finished" podID="407d3a7d-bc65-4626-8c49-3cb5091c32c4" containerID="9dbacd0fe02c38a82e961dc84897d35f964bafbe48d80c4df6ecc230190895a2" exitCode=0
Nov 02 01:48:32.712138 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.712030    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"
Nov 02 01:48:32.712138 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.712036    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589" event={"ID":"407d3a7d-bc65-4626-8c49-3cb5091c32c4","Type":"ContainerDied","Data":"9dbacd0fe02c38a82e961dc84897d35f964bafbe48d80c4df6ecc230190895a2"}
Nov 02 01:48:32.712138 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.712118    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589" event={"ID":"407d3a7d-bc65-4626-8c49-3cb5091c32c4","Type":"ContainerDied","Data":"58ef2d6418582dc2bb3f07da8bca30fb3e4ade835b10da77d006bda97a3a720e"}
Nov 02 01:48:32.712352 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.712149    5875 scope.go:122] "RemoveContainer" containerID="9dbacd0fe02c38a82e961dc84897d35f964bafbe48d80c4df6ecc230190895a2"
Nov 02 01:48:32.721580 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.721562    5875 scope.go:122] "RemoveContainer" containerID="9dbacd0fe02c38a82e961dc84897d35f964bafbe48d80c4df6ecc230190895a2"
Nov 02 01:48:32.721855 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:48:32.721832    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dbacd0fe02c38a82e961dc84897d35f964bafbe48d80c4df6ecc230190895a2\": container with ID starting with 9dbacd0fe02c38a82e961dc84897d35f964bafbe48d80c4df6ecc230190895a2 not found: ID does not exist" containerID="9dbacd0fe02c38a82e961dc84897d35f964bafbe48d80c4df6ecc230190895a2"
Nov 02 01:48:32.721914 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.721867    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dbacd0fe02c38a82e961dc84897d35f964bafbe48d80c4df6ecc230190895a2"} err="failed to get container status \"9dbacd0fe02c38a82e961dc84897d35f964bafbe48d80c4df6ecc230190895a2\": rpc error: code = NotFound desc = could not find container \"9dbacd0fe02c38a82e961dc84897d35f964bafbe48d80c4df6ecc230190895a2\": container with ID starting with 9dbacd0fe02c38a82e961dc84897d35f964bafbe48d80c4df6ecc230190895a2 not found: ID does not exist"
Nov 02 01:48:32.925922 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.925889    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"]
Nov 02 01:48:32.975664 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:32.975637    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-rd589"]
Nov 02 01:48:33.129042 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:33.129012    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:48:34.694018 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:34.693984    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="407d3a7d-bc65-4626-8c49-3cb5091c32c4" path="/var/lib/kubelet/pods/407d3a7d-bc65-4626-8c49-3cb5091c32c4/volumes"
Nov 02 01:48:35.709284 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.709251    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"]
Nov 02 01:48:35.709813 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.709800    5875 cpu_manager.go:414] "containerMap: removing container" podUID="407d3a7d-bc65-4626-8c49-3cb5091c32c4" containerName="istio-proxy"
Nov 02 01:48:35.709856 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.709814    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="407d3a7d-bc65-4626-8c49-3cb5091c32c4" containerName="istio-proxy"
Nov 02 01:48:35.709856 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.709830    5875 cpu_manager.go:414] "containerMap: removing container" podUID="dcd89e9b-403f-41e7-889b-615ebe4e5b24" containerName="revoke-keys"
Nov 02 01:48:35.709856 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.709836    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcd89e9b-403f-41e7-889b-615ebe4e5b24" containerName="revoke-keys"
Nov 02 01:48:35.709943 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.709917    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="407d3a7d-bc65-4626-8c49-3cb5091c32c4" containerName="istio-proxy"
Nov 02 01:48:35.709943 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.709926    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcd89e9b-403f-41e7-889b-615ebe4e5b24" containerName="revoke-keys"
Nov 02 01:48:35.714506 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.714487    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.766607 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.766585    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ed0600-openshift-default-dockercfg-dhrkb\""
Nov 02 01:48:35.783116 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.783088    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-credential-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.783237 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.783117    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f87d2fb1-742f-4632-bd55-5710c177b1f6-istiod-ca-cert\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.783237 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.783144    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-podinfo\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.783334 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.783288    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-envoy\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.783334 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.783318    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-workload-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.783407 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.783336    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-token\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.783407 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.783359    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24qkj\" (UniqueName: \"kubernetes.io/projected/f87d2fb1-742f-4632-bd55-5710c177b1f6-kube-api-access-24qkj\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.783407 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.783383    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-workload-certs\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.783536 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.783518    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-data\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.819803 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.819778    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"]
Nov 02 01:48:35.884585 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.884553    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-credential-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.884585 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.884585    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f87d2fb1-742f-4632-bd55-5710c177b1f6-istiod-ca-cert\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.884817 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.884616    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-podinfo\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.884817 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.884653    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-envoy\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.884817 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.884684    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-workload-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.884817 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.884705    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-token\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.884817 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.884732    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24qkj\" (UniqueName: \"kubernetes.io/projected/f87d2fb1-742f-4632-bd55-5710c177b1f6-kube-api-access-24qkj\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.884817 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.884781    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-workload-certs\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.885135 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.884864    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-data\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.885135 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.884987    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-credential-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.885661 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.885397    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-workload-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.885661 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.885618    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-workload-certs\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.885955 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.885929    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-data\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.885955 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.885948    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f87d2fb1-742f-4632-bd55-5710c177b1f6-istiod-ca-cert\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.887669 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.887643    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-envoy\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.888085 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.888063    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-podinfo\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.950324 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.950298    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24qkj\" (UniqueName: \"kubernetes.io/projected/f87d2fb1-742f-4632-bd55-5710c177b1f6-kube-api-access-24qkj\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:35.951124 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:35.951103    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-token\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:36.028181 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:36.028117    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:36.455020 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:36.454989    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"]
Nov 02 01:48:36.456708 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:48:36.456680    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf87d2fb1_742f_4632_bd55_5710c177b1f6.slice/crio-d0d21d9448b2580f23abe0184fff9e823999866af46735114f9530d01991506a WatchSource:0}: Error finding container d0d21d9448b2580f23abe0184fff9e823999866af46735114f9530d01991506a: Status 404 returned error can't find the container with id d0d21d9448b2580f23abe0184fff9e823999866af46735114f9530d01991506a
Nov 02 01:48:36.459078 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:36.459047    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:48:36.459131 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:36.459108    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:48:36.459164 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:36.459137    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:48:36.739536 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:36.739483    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg" event={"ID":"f87d2fb1-742f-4632-bd55-5710c177b1f6","Type":"ContainerStarted","Data":"ad335955b578b7b7cbdf96675771b01a6ec17c90aa13782b3c6118b004819c28"}
Nov 02 01:48:36.739919 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:36.739555    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg" event={"ID":"f87d2fb1-742f-4632-bd55-5710c177b1f6","Type":"ContainerStarted","Data":"d0d21d9448b2580f23abe0184fff9e823999866af46735114f9530d01991506a"}
Nov 02 01:48:36.912523 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:36.912468    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg" podStartSLOduration=1.9124518350000002 podStartE2EDuration="1.912451835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:36.861276918 +0000 UTC m=+964.302313416" watchObservedRunningTime="2026-11-02 01:48:36.912451835 +0000 UTC m=+964.353488388"
Nov 02 01:48:37.029460 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.029375    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:37.031139 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.031110    5875 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 01:48:37.031275 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.031161    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg" podUID="f87d2fb1-742f-4632-bd55-5710c177b1f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 01:48:37.314991 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.314875    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"]
Nov 02 01:48:37.326271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.326245    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.378622 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.378596    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-55bc84-openshift-default-dockercfg-7g4tr\""
Nov 02 01:48:37.420251 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.408184    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"]
Nov 02 01:48:37.428268 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.426908    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.428268 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.427122    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.428268 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.427181    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.428268 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.427206    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bceaf892-6e4e-4472-bf8d-34b1fe161036-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.428268 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.427355    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.428268 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.427400    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.428268 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.427430    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.428268 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.427468    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzxmz\" (UniqueName: \"kubernetes.io/projected/bceaf892-6e4e-4472-bf8d-34b1fe161036-kube-api-access-pzxmz\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.428268 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.427506    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.528987 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.528951    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.528987 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.528995    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.529211 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.529025    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.529211 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.529172    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bceaf892-6e4e-4472-bf8d-34b1fe161036-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.529368 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.529290    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.529368 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.529336    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.529465 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.529368    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.529465 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.529408    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzxmz\" (UniqueName: \"kubernetes.io/projected/bceaf892-6e4e-4472-bf8d-34b1fe161036-kube-api-access-pzxmz\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.529465 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.529449    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.529604 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.529499    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.529604 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.529446    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.529854 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.529785    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.529854 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.529804    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.530011 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.529872    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bceaf892-6e4e-4472-bf8d-34b1fe161036-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.532048 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.532024    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.532176 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.532157    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.592934 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.592868    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.592934 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.592925    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzxmz\" (UniqueName: \"kubernetes.io/projected/bceaf892-6e4e-4472-bf8d-34b1fe161036-kube-api-access-pzxmz\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:37.641985 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:37.641952    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:38.029568 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:38.029537    5875 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 01:48:38.029967 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:38.029600    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg" podUID="f87d2fb1-742f-4632-bd55-5710c177b1f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 01:48:38.093963 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:48:38.093938    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbceaf892_6e4e_4472_bf8d_34b1fe161036.slice/crio-a816bcb373c4eebe2a1bd1fea6616627d557e783c60543936143616010a2983b WatchSource:0}: Error finding container a816bcb373c4eebe2a1bd1fea6616627d557e783c60543936143616010a2983b: Status 404 returned error can't find the container with id a816bcb373c4eebe2a1bd1fea6616627d557e783c60543936143616010a2983b
Nov 02 01:48:38.096923 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:38.096887    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:48:38.097003 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:38.096977    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:48:38.097037 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:38.097020    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:48:38.132277 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:38.132250    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"]
Nov 02 01:48:38.770129 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:38.770095    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw" event={"ID":"bceaf892-6e4e-4472-bf8d-34b1fe161036","Type":"ContainerStarted","Data":"867fc8e3aacab21720f28a83a60d87a1999b0b1cb75cf4a6e7c6ae051b997ce7"}
Nov 02 01:48:38.770343 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:38.770141    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw" event={"ID":"bceaf892-6e4e-4472-bf8d-34b1fe161036","Type":"ContainerStarted","Data":"a816bcb373c4eebe2a1bd1fea6616627d557e783c60543936143616010a2983b"}
Nov 02 01:48:38.948674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:38.948600    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw" podStartSLOduration=1.948574185 podStartE2EDuration="1.948574185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:38.891754685 +0000 UTC m=+966.332791194" watchObservedRunningTime="2026-11-02 01:48:38.948574185 +0000 UTC m=+966.389610684"
Nov 02 01:48:39.028809 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:39.028714    5875 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 01:48:39.029060 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:39.029034    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg" podUID="f87d2fb1-742f-4632-bd55-5710c177b1f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 01:48:39.647346 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:39.647307    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:39.648048 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:39.647393    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 02 01:48:39.648258 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:39.648209    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw" podUID="bceaf892-6e4e-4472-bf8d-34b1fe161036" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 02 01:48:40.028668 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:40.028630    5875 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 01:48:40.028964 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:40.028936    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg" podUID="f87d2fb1-742f-4632-bd55-5710c177b1f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 01:48:40.645247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:40.644585    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 02 01:48:40.645247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:40.644650    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw" podUID="bceaf892-6e4e-4472-bf8d-34b1fe161036" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 02 01:48:41.045245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:41.031742    5875 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 01:48:41.045245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:41.031818    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg" podUID="f87d2fb1-742f-4632-bd55-5710c177b1f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 01:48:41.370880 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:48:41.370685    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:48:41.643699 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:41.643613    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 02 01:48:41.643955 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:41.643928    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw" podUID="bceaf892-6e4e-4472-bf8d-34b1fe161036" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 02 01:48:42.029422 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:42.029382    5875 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 01:48:42.029749 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:42.029719    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg" podUID="f87d2fb1-742f-4632-bd55-5710c177b1f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 01:48:42.642478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:42.642436    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 02 01:48:42.642957 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:42.642497    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw" podUID="bceaf892-6e4e-4472-bf8d-34b1fe161036" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 02 01:48:43.295439 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:43.295411    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:43.295826 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:43.295794    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:43.296700 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:43.296682    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 01:48:43.645811 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:43.645342    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 02 01:48:43.645811 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:43.645401    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw" podUID="bceaf892-6e4e-4472-bf8d-34b1fe161036" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 02 01:48:44.642842 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:44.642809    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 02 01:48:44.643027 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:44.642863    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw" podUID="bceaf892-6e4e-4472-bf8d-34b1fe161036" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 02 01:48:45.646442 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:45.646409    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:45.646963 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:45.646944    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:45.647652 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:45.647632    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 01:48:56.868893 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:56.868856    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5892afd2-pnwph"]
Nov 02 01:48:56.873028 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:56.873012    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5892afd2-pnwph"
Nov 02 01:48:56.926870 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:56.926845    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:48:56.932513 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:56.932484    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5892afd2-pnwph"]
Nov 02 01:48:56.976418 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:56.976394    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:48:56.998053 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:56.998024    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323f5225-14c7-48a2-bf86-097cbb97afb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5892afd2-pnwph\" (UID: \"323f5225-14c7-48a2-bf86-097cbb97afb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5892afd2-pnwph"
Nov 02 01:48:56.998337 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:56.998318    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323f5225-14c7-48a2-bf86-097cbb97afb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5892afd2-pnwph\" (UID: \"323f5225-14c7-48a2-bf86-097cbb97afb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5892afd2-pnwph"
Nov 02 01:48:56.998473 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:56.998450    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4xnn\" (UniqueName: \"kubernetes.io/projected/323f5225-14c7-48a2-bf86-097cbb97afb9-kube-api-access-c4xnn\") pod \"maas-api-revoke-sub-5892afd2-pnwph\" (UID: \"323f5225-14c7-48a2-bf86-097cbb97afb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5892afd2-pnwph"
Nov 02 01:48:57.099813 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:57.099773    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323f5225-14c7-48a2-bf86-097cbb97afb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5892afd2-pnwph\" (UID: \"323f5225-14c7-48a2-bf86-097cbb97afb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5892afd2-pnwph"
Nov 02 01:48:57.099993 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:57.099852    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4xnn\" (UniqueName: \"kubernetes.io/projected/323f5225-14c7-48a2-bf86-097cbb97afb9-kube-api-access-c4xnn\") pod \"maas-api-revoke-sub-5892afd2-pnwph\" (UID: \"323f5225-14c7-48a2-bf86-097cbb97afb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5892afd2-pnwph"
Nov 02 01:48:57.099993 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:57.099900    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323f5225-14c7-48a2-bf86-097cbb97afb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5892afd2-pnwph\" (UID: \"323f5225-14c7-48a2-bf86-097cbb97afb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5892afd2-pnwph"
Nov 02 01:48:57.100783 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:57.100760    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323f5225-14c7-48a2-bf86-097cbb97afb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5892afd2-pnwph\" (UID: \"323f5225-14c7-48a2-bf86-097cbb97afb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5892afd2-pnwph"
Nov 02 01:48:57.103030 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:57.103006    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323f5225-14c7-48a2-bf86-097cbb97afb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5892afd2-pnwph\" (UID: \"323f5225-14c7-48a2-bf86-097cbb97afb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5892afd2-pnwph"
Nov 02 01:48:57.161145 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:57.161074    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4xnn\" (UniqueName: \"kubernetes.io/projected/323f5225-14c7-48a2-bf86-097cbb97afb9-kube-api-access-c4xnn\") pod \"maas-api-revoke-sub-5892afd2-pnwph\" (UID: \"323f5225-14c7-48a2-bf86-097cbb97afb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5892afd2-pnwph"
Nov 02 01:48:57.186824 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:57.186800    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5892afd2-pnwph"
Nov 02 01:48:57.560821 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:57.560793    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5892afd2-pnwph"]
Nov 02 01:48:57.562508 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:48:57.562485    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323f5225_14c7_48a2_bf86_097cbb97afb9.slice/crio-63d3299bba15326579676f41d1c1d1f16e4726cf61c905d847af3fafe87942dc WatchSource:0}: Error finding container 63d3299bba15326579676f41d1c1d1f16e4726cf61c905d847af3fafe87942dc: Status 404 returned error can't find the container with id 63d3299bba15326579676f41d1c1d1f16e4726cf61c905d847af3fafe87942dc
Nov 02 01:48:57.911835 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:57.911761    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5892afd2-pnwph" event={"ID":"323f5225-14c7-48a2-bf86-097cbb97afb9","Type":"ContainerStarted","Data":"363089c5ff35f72ae2a7c72aceaa6f14881fff146bcf2c272ba17d9a3256f01b"}
Nov 02 01:48:57.911835 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:57.911809    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5892afd2-pnwph" event={"ID":"323f5225-14c7-48a2-bf86-097cbb97afb9","Type":"ContainerStarted","Data":"63d3299bba15326579676f41d1c1d1f16e4726cf61c905d847af3fafe87942dc"}
Nov 02 01:48:58.028772 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:58.028711    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5892afd2-pnwph" podStartSLOduration=2.028690548 podStartE2EDuration="2.028690548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:58.027917875 +0000 UTC m=+985.468954409" watchObservedRunningTime="2026-11-02 01:48:58.028690548 +0000 UTC m=+985.469727047"
Nov 02 01:48:58.919473 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:58.919440    5875 generic.go:356] "Generic (PLEG): container finished" podID="323f5225-14c7-48a2-bf86-097cbb97afb9" containerID="363089c5ff35f72ae2a7c72aceaa6f14881fff146bcf2c272ba17d9a3256f01b" exitCode=0
Nov 02 01:48:58.919851 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:48:58.919543    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5892afd2-pnwph" event={"ID":"323f5225-14c7-48a2-bf86-097cbb97afb9","Type":"ContainerDied","Data":"363089c5ff35f72ae2a7c72aceaa6f14881fff146bcf2c272ba17d9a3256f01b"}
Nov 02 01:49:00.071473 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:00.071445    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5892afd2-pnwph"
Nov 02 01:49:00.136094 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:00.136063    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/323f5225-14c7-48a2-bf86-097cbb97afb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323f5225-14c7-48a2-bf86-097cbb97afb9-maas-api-xks-ca\") pod \"323f5225-14c7-48a2-bf86-097cbb97afb9\" (UID: \"323f5225-14c7-48a2-bf86-097cbb97afb9\") "
Nov 02 01:49:00.136286 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:00.136174    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/323f5225-14c7-48a2-bf86-097cbb97afb9-kube-api-access-c4xnn\" (UniqueName: \"kubernetes.io/projected/323f5225-14c7-48a2-bf86-097cbb97afb9-kube-api-access-c4xnn\") pod \"323f5225-14c7-48a2-bf86-097cbb97afb9\" (UID: \"323f5225-14c7-48a2-bf86-097cbb97afb9\") "
Nov 02 01:49:00.136286 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:00.136247    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/323f5225-14c7-48a2-bf86-097cbb97afb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323f5225-14c7-48a2-bf86-097cbb97afb9-maas-api-openshift-ca\") pod \"323f5225-14c7-48a2-bf86-097cbb97afb9\" (UID: \"323f5225-14c7-48a2-bf86-097cbb97afb9\") "
Nov 02 01:49:00.136594 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:00.136569    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/323f5225-14c7-48a2-bf86-097cbb97afb9-maas-api-openshift-ca" pod "323f5225-14c7-48a2-bf86-097cbb97afb9" (UID: "323f5225-14c7-48a2-bf86-097cbb97afb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:00.138374 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:00.138348    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323f5225-14c7-48a2-bf86-097cbb97afb9-kube-api-access-c4xnn" pod "323f5225-14c7-48a2-bf86-097cbb97afb9" (UID: "323f5225-14c7-48a2-bf86-097cbb97afb9"). InnerVolumeSpecName "kube-api-access-c4xnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:00.138620 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:00.138596    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/323f5225-14c7-48a2-bf86-097cbb97afb9-maas-api-xks-ca" pod "323f5225-14c7-48a2-bf86-097cbb97afb9" (UID: "323f5225-14c7-48a2-bf86-097cbb97afb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:00.237381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:00.237347    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4xnn\" (UniqueName: \"kubernetes.io/projected/323f5225-14c7-48a2-bf86-097cbb97afb9-kube-api-access-c4xnn\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:00.237381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:00.237376    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323f5225-14c7-48a2-bf86-097cbb97afb9-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:00.237564 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:00.237390    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323f5225-14c7-48a2-bf86-097cbb97afb9-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:00.931627 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:00.931597    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5892afd2-pnwph"
Nov 02 01:49:00.931801 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:00.931642    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5892afd2-pnwph" event={"ID":"323f5225-14c7-48a2-bf86-097cbb97afb9","Type":"ContainerDied","Data":"63d3299bba15326579676f41d1c1d1f16e4726cf61c905d847af3fafe87942dc"}
Nov 02 01:49:00.931801 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:00.931717    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63d3299bba15326579676f41d1c1d1f16e4726cf61c905d847af3fafe87942dc"
Nov 02 01:49:11.319712 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:49:11.319605    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:49:12.201422 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.201387    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"]
Nov 02 01:49:12.202097 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.202079    5875 cpu_manager.go:414] "containerMap: removing container" podUID="323f5225-14c7-48a2-bf86-097cbb97afb9" containerName="revoke-keys"
Nov 02 01:49:12.202150 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.202100    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="323f5225-14c7-48a2-bf86-097cbb97afb9" containerName="revoke-keys"
Nov 02 01:49:12.202247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.202235    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="323f5225-14c7-48a2-bf86-097cbb97afb9" containerName="revoke-keys"
Nov 02 01:49:12.206891 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.206873    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.280892 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.280852    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.281086 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.280914    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.281086 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.280995    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.281086 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.281074    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.281299 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.281138    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.281299 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.281154    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pt4w\" (UniqueName: \"kubernetes.io/projected/e1d7b68a-92f5-47c6-bedb-d75802e2232d-kube-api-access-7pt4w\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.281299 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.281198    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.281299 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.281275    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.281513 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.281467    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.318480 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.318450    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"]
Nov 02 01:49:12.322710 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.322687    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74fdb954-fm82f"]
Nov 02 01:49:12.326839 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.326823    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fdb954-fm82f"
Nov 02 01:49:12.381605 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.381567    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:49:12.381843 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.381814    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:49:12.382352 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.382330    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.382452 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.382379    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.382452 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.382406    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.382452 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.382448    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.382608 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.382548    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.382664 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.382631    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pt4w\" (UniqueName: \"kubernetes.io/projected/e1d7b68a-92f5-47c6-bedb-d75802e2232d-kube-api-access-7pt4w\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.382710 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.382688    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.382761 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.382740    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.382830 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.382814    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.382905 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.382885    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.383911 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.383191    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.383911 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.382838    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.383911 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.383466    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.383911 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.383855    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.385023 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.384998    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74fdb954-fm82f"]
Nov 02 01:49:12.386653 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.386627    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.386766 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.386722    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.450278 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.448971    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.450278 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.449263    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pt4w\" (UniqueName: \"kubernetes.io/projected/e1d7b68a-92f5-47c6-bedb-d75802e2232d-kube-api-access-7pt4w\") pod \"maas-default-gateway-openshift-default-778b7b5c47-dthb5\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.483775 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.483749    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/709291d6-23c8-4d12-8285-6a2630fa1e7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74fdb954-fm82f\" (UID: \"709291d6-23c8-4d12-8285-6a2630fa1e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fdb954-fm82f"
Nov 02 01:49:12.483946 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.483931    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/709291d6-23c8-4d12-8285-6a2630fa1e7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74fdb954-fm82f\" (UID: \"709291d6-23c8-4d12-8285-6a2630fa1e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fdb954-fm82f"
Nov 02 01:49:12.483987 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.483955    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvcmc\" (UniqueName: \"kubernetes.io/projected/709291d6-23c8-4d12-8285-6a2630fa1e7c-kube-api-access-qvcmc\") pod \"maas-api-revoke-sub-74fdb954-fm82f\" (UID: \"709291d6-23c8-4d12-8285-6a2630fa1e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fdb954-fm82f"
Nov 02 01:49:12.522357 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.522324    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:12.592263 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.586548    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/709291d6-23c8-4d12-8285-6a2630fa1e7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74fdb954-fm82f\" (UID: \"709291d6-23c8-4d12-8285-6a2630fa1e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fdb954-fm82f"
Nov 02 01:49:12.592263 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.587346    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/709291d6-23c8-4d12-8285-6a2630fa1e7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74fdb954-fm82f\" (UID: \"709291d6-23c8-4d12-8285-6a2630fa1e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fdb954-fm82f"
Nov 02 01:49:12.592263 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.587433    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvcmc\" (UniqueName: \"kubernetes.io/projected/709291d6-23c8-4d12-8285-6a2630fa1e7c-kube-api-access-qvcmc\") pod \"maas-api-revoke-sub-74fdb954-fm82f\" (UID: \"709291d6-23c8-4d12-8285-6a2630fa1e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fdb954-fm82f"
Nov 02 01:49:12.592263 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.587990    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/709291d6-23c8-4d12-8285-6a2630fa1e7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74fdb954-fm82f\" (UID: \"709291d6-23c8-4d12-8285-6a2630fa1e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fdb954-fm82f"
Nov 02 01:49:12.592585 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.592477    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/709291d6-23c8-4d12-8285-6a2630fa1e7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74fdb954-fm82f\" (UID: \"709291d6-23c8-4d12-8285-6a2630fa1e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fdb954-fm82f"
Nov 02 01:49:12.648274 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.648212    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvcmc\" (UniqueName: \"kubernetes.io/projected/709291d6-23c8-4d12-8285-6a2630fa1e7c-kube-api-access-qvcmc\") pod \"maas-api-revoke-sub-74fdb954-fm82f\" (UID: \"709291d6-23c8-4d12-8285-6a2630fa1e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fdb954-fm82f"
Nov 02 01:49:12.941868 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:12.941794    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fdb954-fm82f"
Nov 02 01:49:13.002374 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:13.002149    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"]
Nov 02 01:49:13.010452 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:13.008749    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:49:13.010452 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:13.008835    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:49:13.010452 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:13.008900    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:49:13.320916 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:13.319819    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74fdb954-fm82f"]
Nov 02 01:49:13.321896 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:49:13.321869    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod709291d6_23c8_4d12_8285_6a2630fa1e7c.slice/crio-b2721b7edff3935f0ca6eb912e11ab9a84b172c688d85ec5513453d53a9e0100 WatchSource:0}: Error finding container b2721b7edff3935f0ca6eb912e11ab9a84b172c688d85ec5513453d53a9e0100: Status 404 returned error can't find the container with id b2721b7edff3935f0ca6eb912e11ab9a84b172c688d85ec5513453d53a9e0100
Nov 02 01:49:14.026753 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:14.026496    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5" event={"ID":"e1d7b68a-92f5-47c6-bedb-d75802e2232d","Type":"ContainerStarted","Data":"940bc71f358809f78adf665b284dcf6feefd4b3c3e6490e71c83c493a11008cf"}
Nov 02 01:49:14.026753 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:14.026564    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5" event={"ID":"e1d7b68a-92f5-47c6-bedb-d75802e2232d","Type":"ContainerStarted","Data":"cac892884e06abadbd425a42d0035ed9c27cbd89e6d3c36b7e0baf136f0209f1"}
Nov 02 01:49:14.028626 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:14.028602    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fdb954-fm82f" event={"ID":"709291d6-23c8-4d12-8285-6a2630fa1e7c","Type":"ContainerStarted","Data":"2eeae924d0c55f636e854230054e10b6f2545c1e7f9b698f854f17f0000b53b0"}
Nov 02 01:49:14.028784 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:14.028638    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fdb954-fm82f" event={"ID":"709291d6-23c8-4d12-8285-6a2630fa1e7c","Type":"ContainerStarted","Data":"b2721b7edff3935f0ca6eb912e11ab9a84b172c688d85ec5513453d53a9e0100"}
Nov 02 01:49:14.225346 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:14.223091    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5" podStartSLOduration=2.223069813 podStartE2EDuration="2.223069813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:14.1988456 +0000 UTC m=+1001.639882097" watchObservedRunningTime="2026-11-02 01:49:14.223069813 +0000 UTC m=+1001.664106311"
Nov 02 01:49:14.318373 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:14.318253    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fdb954-fm82f" podStartSLOduration=2.318234326 podStartE2EDuration="2.318234326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:14.316648175 +0000 UTC m=+1001.757684674" watchObservedRunningTime="2026-11-02 01:49:14.318234326 +0000 UTC m=+1001.759270815"
Nov 02 01:49:14.523980 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:14.523896    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:14.525252 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:14.524916    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-dthb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 01:49:14.525252 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:14.524970    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5" podUID="e1d7b68a-92f5-47c6-bedb-d75802e2232d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 01:49:15.037379 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:15.037348    5875 generic.go:356] "Generic (PLEG): container finished" podID="709291d6-23c8-4d12-8285-6a2630fa1e7c" containerID="2eeae924d0c55f636e854230054e10b6f2545c1e7f9b698f854f17f0000b53b0" exitCode=0
Nov 02 01:49:15.037852 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:15.037504    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fdb954-fm82f" event={"ID":"709291d6-23c8-4d12-8285-6a2630fa1e7c","Type":"ContainerDied","Data":"2eeae924d0c55f636e854230054e10b6f2545c1e7f9b698f854f17f0000b53b0"}
Nov 02 01:49:15.524169 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:15.523817    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-dthb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 01:49:15.524169 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:15.523877    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5" podUID="e1d7b68a-92f5-47c6-bedb-d75802e2232d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 01:49:15.629057 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:15.629019    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:49:16.357293 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:16.354385    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fdb954-fm82f"
Nov 02 01:49:16.466275 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:16.464714    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/709291d6-23c8-4d12-8285-6a2630fa1e7c-kube-api-access-qvcmc\" (UniqueName: \"kubernetes.io/projected/709291d6-23c8-4d12-8285-6a2630fa1e7c-kube-api-access-qvcmc\") pod \"709291d6-23c8-4d12-8285-6a2630fa1e7c\" (UID: \"709291d6-23c8-4d12-8285-6a2630fa1e7c\") "
Nov 02 01:49:16.466275 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:16.464793    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/709291d6-23c8-4d12-8285-6a2630fa1e7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/709291d6-23c8-4d12-8285-6a2630fa1e7c-maas-api-xks-ca\") pod \"709291d6-23c8-4d12-8285-6a2630fa1e7c\" (UID: \"709291d6-23c8-4d12-8285-6a2630fa1e7c\") "
Nov 02 01:49:16.466275 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:16.464906    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/709291d6-23c8-4d12-8285-6a2630fa1e7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/709291d6-23c8-4d12-8285-6a2630fa1e7c-maas-api-openshift-ca\") pod \"709291d6-23c8-4d12-8285-6a2630fa1e7c\" (UID: \"709291d6-23c8-4d12-8285-6a2630fa1e7c\") "
Nov 02 01:49:16.466588 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:16.466282    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/709291d6-23c8-4d12-8285-6a2630fa1e7c-maas-api-openshift-ca" pod "709291d6-23c8-4d12-8285-6a2630fa1e7c" (UID: "709291d6-23c8-4d12-8285-6a2630fa1e7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:16.487441 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:16.487339    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709291d6-23c8-4d12-8285-6a2630fa1e7c-kube-api-access-qvcmc" pod "709291d6-23c8-4d12-8285-6a2630fa1e7c" (UID: "709291d6-23c8-4d12-8285-6a2630fa1e7c"). InnerVolumeSpecName "kube-api-access-qvcmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:16.501285 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:16.490675    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/709291d6-23c8-4d12-8285-6a2630fa1e7c-maas-api-xks-ca" pod "709291d6-23c8-4d12-8285-6a2630fa1e7c" (UID: "709291d6-23c8-4d12-8285-6a2630fa1e7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:16.555244 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:16.553344    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-dthb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 01:49:16.555244 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:16.553409    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5" podUID="e1d7b68a-92f5-47c6-bedb-d75802e2232d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 01:49:16.570779 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:16.568620    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/709291d6-23c8-4d12-8285-6a2630fa1e7c-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:16.570779 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:16.568661    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvcmc\" (UniqueName: \"kubernetes.io/projected/709291d6-23c8-4d12-8285-6a2630fa1e7c-kube-api-access-qvcmc\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:16.570779 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:16.568678    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/709291d6-23c8-4d12-8285-6a2630fa1e7c-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:16.932109 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:49:16.932073    5875 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod709291d6_23c8_4d12_8285_6a2630fa1e7c.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 01:49:17.060405 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:17.060361    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fdb954-fm82f" event={"ID":"709291d6-23c8-4d12-8285-6a2630fa1e7c","Type":"ContainerDied","Data":"b2721b7edff3935f0ca6eb912e11ab9a84b172c688d85ec5513453d53a9e0100"}
Nov 02 01:49:17.060606 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:17.060422    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2721b7edff3935f0ca6eb912e11ab9a84b172c688d85ec5513453d53a9e0100"
Nov 02 01:49:17.060606 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:17.060452    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fdb954-fm82f"
Nov 02 01:49:17.524246 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:17.523859    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-dthb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 01:49:17.524246 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:17.523933    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5" podUID="e1d7b68a-92f5-47c6-bedb-d75802e2232d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 01:49:18.524248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:18.523730    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-dthb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 01:49:18.524248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:18.523786    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5" podUID="e1d7b68a-92f5-47c6-bedb-d75802e2232d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 01:49:19.522743 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:19.522707    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-dthb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 01:49:19.522928 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:19.522760    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5" podUID="e1d7b68a-92f5-47c6-bedb-d75802e2232d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 01:49:19.697055 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:19.697014    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97ffaad1-7m4qt"]
Nov 02 01:49:19.697590 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:19.697576    5875 cpu_manager.go:414] "containerMap: removing container" podUID="709291d6-23c8-4d12-8285-6a2630fa1e7c" containerName="revoke-keys"
Nov 02 01:49:19.697634 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:19.697593    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="709291d6-23c8-4d12-8285-6a2630fa1e7c" containerName="revoke-keys"
Nov 02 01:49:19.697707 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:19.697698    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="709291d6-23c8-4d12-8285-6a2630fa1e7c" containerName="revoke-keys"
Nov 02 01:49:19.701186 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:19.701169    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ffaad1-7m4qt"
Nov 02 01:49:19.755537 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:19.755507    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:49:19.759699 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:19.759673    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97ffaad1-7m4qt"]
Nov 02 01:49:19.804837 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:19.804761    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:49:19.822877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:19.822848    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d4c395d-b06a-460f-aabb-d5395e5293cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97ffaad1-7m4qt\" (UID: \"0d4c395d-b06a-460f-aabb-d5395e5293cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ffaad1-7m4qt"
Nov 02 01:49:19.823019 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:19.822926    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvsfz\" (UniqueName: \"kubernetes.io/projected/0d4c395d-b06a-460f-aabb-d5395e5293cf-kube-api-access-jvsfz\") pod \"maas-api-revoke-sub-97ffaad1-7m4qt\" (UID: \"0d4c395d-b06a-460f-aabb-d5395e5293cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ffaad1-7m4qt"
Nov 02 01:49:19.823118 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:19.823098    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d4c395d-b06a-460f-aabb-d5395e5293cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97ffaad1-7m4qt\" (UID: \"0d4c395d-b06a-460f-aabb-d5395e5293cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ffaad1-7m4qt"
Nov 02 01:49:19.924318 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:19.924292    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvsfz\" (UniqueName: \"kubernetes.io/projected/0d4c395d-b06a-460f-aabb-d5395e5293cf-kube-api-access-jvsfz\") pod \"maas-api-revoke-sub-97ffaad1-7m4qt\" (UID: \"0d4c395d-b06a-460f-aabb-d5395e5293cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ffaad1-7m4qt"
Nov 02 01:49:19.924449 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:19.924384    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d4c395d-b06a-460f-aabb-d5395e5293cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97ffaad1-7m4qt\" (UID: \"0d4c395d-b06a-460f-aabb-d5395e5293cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ffaad1-7m4qt"
Nov 02 01:49:19.924449 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:19.924443    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d4c395d-b06a-460f-aabb-d5395e5293cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97ffaad1-7m4qt\" (UID: \"0d4c395d-b06a-460f-aabb-d5395e5293cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ffaad1-7m4qt"
Nov 02 01:49:19.925004 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:19.924983    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d4c395d-b06a-460f-aabb-d5395e5293cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97ffaad1-7m4qt\" (UID: \"0d4c395d-b06a-460f-aabb-d5395e5293cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ffaad1-7m4qt"
Nov 02 01:49:19.926822 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:19.926798    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d4c395d-b06a-460f-aabb-d5395e5293cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97ffaad1-7m4qt\" (UID: \"0d4c395d-b06a-460f-aabb-d5395e5293cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ffaad1-7m4qt"
Nov 02 01:49:19.986792 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:19.986767    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvsfz\" (UniqueName: \"kubernetes.io/projected/0d4c395d-b06a-460f-aabb-d5395e5293cf-kube-api-access-jvsfz\") pod \"maas-api-revoke-sub-97ffaad1-7m4qt\" (UID: \"0d4c395d-b06a-460f-aabb-d5395e5293cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ffaad1-7m4qt"
Nov 02 01:49:20.014649 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:20.014622    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ffaad1-7m4qt"
Nov 02 01:49:20.389688 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:20.389659    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97ffaad1-7m4qt"]
Nov 02 01:49:20.392607 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:49:20.392580    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4c395d_b06a_460f_aabb_d5395e5293cf.slice/crio-d1540eaef3df525ec0ca707f4fee148d1c15d1558d09edfe661daeb4a132ca02 WatchSource:0}: Error finding container d1540eaef3df525ec0ca707f4fee148d1c15d1558d09edfe661daeb4a132ca02: Status 404 returned error can't find the container with id d1540eaef3df525ec0ca707f4fee148d1c15d1558d09edfe661daeb4a132ca02
Nov 02 01:49:20.522859 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:20.522828    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-dthb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 01:49:20.523028 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:20.522881    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5" podUID="e1d7b68a-92f5-47c6-bedb-d75802e2232d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 01:49:21.115310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:21.114529    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ffaad1-7m4qt" event={"ID":"0d4c395d-b06a-460f-aabb-d5395e5293cf","Type":"ContainerStarted","Data":"f4520f22c4ea896ea4c27c3df21d555ab9b966d2f2ec61eff80d8afcb7a28c92"}
Nov 02 01:49:21.115310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:21.114593    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ffaad1-7m4qt" event={"ID":"0d4c395d-b06a-460f-aabb-d5395e5293cf","Type":"ContainerStarted","Data":"d1540eaef3df525ec0ca707f4fee148d1c15d1558d09edfe661daeb4a132ca02"}
Nov 02 01:49:21.236975 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:21.236907    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ffaad1-7m4qt" podStartSLOduration=2.236884986 podStartE2EDuration="2.236884986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:21.232258924 +0000 UTC m=+1008.673295416" watchObservedRunningTime="2026-11-02 01:49:21.236884986 +0000 UTC m=+1008.677921484"
Nov 02 01:49:21.979256 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:21.979197    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:21.980425 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:21.980401    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:21.981098 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:21.981077    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:49:22.125027 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:22.124995    5875 generic.go:356] "Generic (PLEG): container finished" podID="0d4c395d-b06a-460f-aabb-d5395e5293cf" containerID="f4520f22c4ea896ea4c27c3df21d555ab9b966d2f2ec61eff80d8afcb7a28c92" exitCode=0
Nov 02 01:49:22.125553 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:22.125074    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ffaad1-7m4qt" event={"ID":"0d4c395d-b06a-460f-aabb-d5395e5293cf","Type":"ContainerDied","Data":"f4520f22c4ea896ea4c27c3df21d555ab9b966d2f2ec61eff80d8afcb7a28c92"}
Nov 02 01:49:23.308764 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:23.308735    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ffaad1-7m4qt"
Nov 02 01:49:23.371190 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:23.371156    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d4c395d-b06a-460f-aabb-d5395e5293cf-kube-api-access-jvsfz\" (UniqueName: \"kubernetes.io/projected/0d4c395d-b06a-460f-aabb-d5395e5293cf-kube-api-access-jvsfz\") pod \"0d4c395d-b06a-460f-aabb-d5395e5293cf\" (UID: \"0d4c395d-b06a-460f-aabb-d5395e5293cf\") "
Nov 02 01:49:23.371385 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:23.371213    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d4c395d-b06a-460f-aabb-d5395e5293cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d4c395d-b06a-460f-aabb-d5395e5293cf-maas-api-xks-ca\") pod \"0d4c395d-b06a-460f-aabb-d5395e5293cf\" (UID: \"0d4c395d-b06a-460f-aabb-d5395e5293cf\") "
Nov 02 01:49:23.371385 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:23.371277    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d4c395d-b06a-460f-aabb-d5395e5293cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d4c395d-b06a-460f-aabb-d5395e5293cf-maas-api-openshift-ca\") pod \"0d4c395d-b06a-460f-aabb-d5395e5293cf\" (UID: \"0d4c395d-b06a-460f-aabb-d5395e5293cf\") "
Nov 02 01:49:23.371692 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:23.371670    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d4c395d-b06a-460f-aabb-d5395e5293cf-maas-api-openshift-ca" pod "0d4c395d-b06a-460f-aabb-d5395e5293cf" (UID: "0d4c395d-b06a-460f-aabb-d5395e5293cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:49:23.373367 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:23.373339    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d4c395d-b06a-460f-aabb-d5395e5293cf-maas-api-xks-ca" pod "0d4c395d-b06a-460f-aabb-d5395e5293cf" (UID: "0d4c395d-b06a-460f-aabb-d5395e5293cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:49:23.373466 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:23.373426    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d4c395d-b06a-460f-aabb-d5395e5293cf-kube-api-access-jvsfz" pod "0d4c395d-b06a-460f-aabb-d5395e5293cf" (UID: "0d4c395d-b06a-460f-aabb-d5395e5293cf"). InnerVolumeSpecName "kube-api-access-jvsfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:49:23.473138 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:23.473104    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvsfz\" (UniqueName: \"kubernetes.io/projected/0d4c395d-b06a-460f-aabb-d5395e5293cf-kube-api-access-jvsfz\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:23.473138 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:23.473133    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d4c395d-b06a-460f-aabb-d5395e5293cf-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:23.473138 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:23.473145    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d4c395d-b06a-460f-aabb-d5395e5293cf-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:49:24.142886 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:24.142855    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ffaad1-7m4qt"
Nov 02 01:49:24.143095 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:24.142900    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97ffaad1-7m4qt" event={"ID":"0d4c395d-b06a-460f-aabb-d5395e5293cf","Type":"ContainerDied","Data":"d1540eaef3df525ec0ca707f4fee148d1c15d1558d09edfe661daeb4a132ca02"}
Nov 02 01:49:24.143095 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:24.142955    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1540eaef3df525ec0ca707f4fee148d1c15d1558d09edfe661daeb4a132ca02"
Nov 02 01:49:34.722529 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:34.722495    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"]
Nov 02 01:49:34.723064 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:34.723051    5875 cpu_manager.go:414] "containerMap: removing container" podUID="0d4c395d-b06a-460f-aabb-d5395e5293cf" containerName="revoke-keys"
Nov 02 01:49:34.723128 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:34.723066    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d4c395d-b06a-460f-aabb-d5395e5293cf" containerName="revoke-keys"
Nov 02 01:49:34.723186 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:34.723176    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d4c395d-b06a-460f-aabb-d5395e5293cf" containerName="revoke-keys"
Nov 02 01:49:34.727625 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:34.727607    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:34.835504 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:34.835472    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"]
Nov 02 01:49:34.904934 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:34.904895    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/752ab42e-3d04-4340-8481-9592789c0367-istio-token\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:34.905114 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:34.904958    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-workload-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:34.905161 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:34.905105    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-credential-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:34.905197 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:34.905170    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-workload-certs\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:34.905262 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:34.905199    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/752ab42e-3d04-4340-8481-9592789c0367-istiod-ca-cert\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:34.905321 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:34.905299    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-istio-data\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:34.905360 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:34.905347    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5482\" (UniqueName: \"kubernetes.io/projected/752ab42e-3d04-4340-8481-9592789c0367-kube-api-access-v5482\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:34.905417 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:34.905403    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/752ab42e-3d04-4340-8481-9592789c0367-istio-podinfo\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:34.905457 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:34.905446    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-istio-envoy\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:35.007063 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:35.006971    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-credential-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:35.007063 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:35.007033    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-workload-certs\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:35.007063 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:35.007062    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/752ab42e-3d04-4340-8481-9592789c0367-istiod-ca-cert\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:35.007384 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:35.007101    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-istio-data\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:35.007384 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:35.007125    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5482\" (UniqueName: \"kubernetes.io/projected/752ab42e-3d04-4340-8481-9592789c0367-kube-api-access-v5482\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:35.007384 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:35.007161    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/752ab42e-3d04-4340-8481-9592789c0367-istio-podinfo\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:35.007384 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:35.007213    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-istio-envoy\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:35.007384 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:35.007325    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/752ab42e-3d04-4340-8481-9592789c0367-istio-token\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:35.007384 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:35.007383    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-workload-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:35.007687 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:35.007420    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-workload-certs\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:35.007687 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:35.007492    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-credential-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:35.007687 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:35.007606    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-istio-data\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:35.007842 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:35.007789    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-workload-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:35.007928 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:35.007909    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/752ab42e-3d04-4340-8481-9592789c0367-istiod-ca-cert\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:35.009611 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:35.009590    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-istio-envoy\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:35.009732 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:35.009713    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/752ab42e-3d04-4340-8481-9592789c0367-istio-podinfo\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:35.068863 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:35.068829    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5482\" (UniqueName: \"kubernetes.io/projected/752ab42e-3d04-4340-8481-9592789c0367-kube-api-access-v5482\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:35.068863 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:35.068847    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/752ab42e-3d04-4340-8481-9592789c0367-istio-token\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:35.342194 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:35.342103    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:35.777739 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:35.777708    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"]
Nov 02 01:49:35.778655 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:49:35.778627    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752ab42e_3d04_4340_8481_9592789c0367.slice/crio-6ce74dfffff3d4f93804f1e187ab9b835216267e410e50342f93505ce8ee1290 WatchSource:0}: Error finding container 6ce74dfffff3d4f93804f1e187ab9b835216267e410e50342f93505ce8ee1290: Status 404 returned error can't find the container with id 6ce74dfffff3d4f93804f1e187ab9b835216267e410e50342f93505ce8ee1290
Nov 02 01:49:35.781306 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:35.781271    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:49:35.781382 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:35.781336    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:49:35.781382 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:35.781367    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:49:36.219631 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:36.219598    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz" event={"ID":"752ab42e-3d04-4340-8481-9592789c0367","Type":"ContainerStarted","Data":"153ec1dfc61d7fa222cc86a5f48663a9b67b0a75fe80349c483e785922ab015d"}
Nov 02 01:49:36.219834 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:36.219661    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz" event={"ID":"752ab42e-3d04-4340-8481-9592789c0367","Type":"ContainerStarted","Data":"6ce74dfffff3d4f93804f1e187ab9b835216267e410e50342f93505ce8ee1290"}
Nov 02 01:49:36.343234 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:36.343202    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:36.344722 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:36.344698    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 01:49:36.344870 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:36.344750    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz" podUID="752ab42e-3d04-4340-8481-9592789c0367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 01:49:36.391798 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:36.391738    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz" podStartSLOduration=2.391723448 podStartE2EDuration="2.391723448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:36.388579708 +0000 UTC m=+1023.829616241" watchObservedRunningTime="2026-11-02 01:49:36.391723448 +0000 UTC m=+1023.832759984"
Nov 02 01:49:37.343289 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:37.343246    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 01:49:37.343686 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:37.343308    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz" podUID="752ab42e-3d04-4340-8481-9592789c0367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 01:49:37.790080 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:37.790038    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"]
Nov 02 01:49:37.794598 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:37.794577    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:37.954731 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:37.954691    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:37.969435 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:37.965209    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:37.969785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:37.969758    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:37.969920 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:37.969905    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:37.970052 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:37.970037    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:37.973278 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:37.970305    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrzj5\" (UniqueName: \"kubernetes.io/projected/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-kube-api-access-xrzj5\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:37.973278 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:37.970342    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:37.973278 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:37.970397    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:37.973278 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:37.970453    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:37.973278 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:37.973258    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"]
Nov 02 01:49:38.046400 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.046324    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"]
Nov 02 01:49:38.054245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.050308    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.073839 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.072579    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrzj5\" (UniqueName: \"kubernetes.io/projected/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-kube-api-access-xrzj5\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:38.073839 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.072631    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:38.073839 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.072683    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:38.073839 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.072722    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-token\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.073839 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.072766    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frchm\" (UniqueName: \"kubernetes.io/projected/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-kube-api-access-frchm\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.073839 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.072812    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:38.073839 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.072863    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-workload-certs\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.073839 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.072992    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-credential-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.073839 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.073041    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:38.073839 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.073137    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-envoy\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.073839 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.073178    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:38.076551 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.074690    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-workload-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.076551 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.074735    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istiod-ca-cert\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.076551 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.074775    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:38.076551 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.074808    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-podinfo\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.076551 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.074838    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:38.076551 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.074895    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-data\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.076551 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.074922    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:38.076551 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.075818    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:38.077425 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.077404    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:38.079382 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.077807    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:38.079382 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.078029    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:38.079877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.079837    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:38.099871 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.095710    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:38.150558 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.150526    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-c62671-openshift-default-dockercfg-nw98n\""
Nov 02 01:49:38.177479 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.161541    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrzj5\" (UniqueName: \"kubernetes.io/projected/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-kube-api-access-xrzj5\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:38.192249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.178489    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-envoy\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.192610 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.192580    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-workload-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.192770 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.192751    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istiod-ca-cert\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.192894 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.192881    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-podinfo\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.193018 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.193005    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-data\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.193213 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.193197    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-token\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.193383 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.193367    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frchm\" (UniqueName: \"kubernetes.io/projected/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-kube-api-access-frchm\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.193537 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.193521    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-workload-certs\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.193666 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.193651    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-credential-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.194021 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.193993    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istiod-ca-cert\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.194160 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.185181    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"]
Nov 02 01:49:38.194326 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.194308    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-credential-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.210256 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.199573    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:38.261310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.241589    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-workload-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.261310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.243538    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-data\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.261310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.243770    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-workload-certs\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.316320 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.292585    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:38.316320 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.293692    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-envoy\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.362145 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.361743    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-token\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.365778 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.363177    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frchm\" (UniqueName: \"kubernetes.io/projected/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-kube-api-access-frchm\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.372065 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.371156    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 01:49:38.372065 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.371208    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz" podUID="752ab42e-3d04-4340-8481-9592789c0367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 01:49:38.378610 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.378557    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-podinfo\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:38.430517 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.430487    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:38.672501 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:38.672137    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:39.354303 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:39.353342    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 01:49:39.354303 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:39.353408    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz" podUID="752ab42e-3d04-4340-8481-9592789c0367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 01:49:39.472336 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:39.472293    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"]
Nov 02 01:49:39.540025 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:39.539997    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"]
Nov 02 01:49:39.541263 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:39.541201    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:49:39.541524 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:39.541498    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:49:39.541707 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:39.541677    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:49:39.565469 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:49:39.565443    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066ee18e_0d0d_4dee_a06c_3c6b218e1a80.slice/crio-36db16eadc5869da0a0bbbb6c72a6121409c7c782ef08affe438a1e3723d0620 WatchSource:0}: Error finding container 36db16eadc5869da0a0bbbb6c72a6121409c7c782ef08affe438a1e3723d0620: Status 404 returned error can't find the container with id 36db16eadc5869da0a0bbbb6c72a6121409c7c782ef08affe438a1e3723d0620
Nov 02 01:49:39.568065 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:39.568021    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:49:39.568171 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:39.568108    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:49:39.568171 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:39.568159    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:49:40.363247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:40.344624    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 01:49:40.363247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:40.344686    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz" podUID="752ab42e-3d04-4340-8481-9592789c0367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 01:49:40.393730 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:40.393686    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k" event={"ID":"066ee18e-0d0d-4dee-a06c-3c6b218e1a80","Type":"ContainerStarted","Data":"75bd4f3dcfaf2f1aa05f5919387416fbac0300660582789cc733bd909cae9811"}
Nov 02 01:49:40.393903 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:40.393784    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k" event={"ID":"066ee18e-0d0d-4dee-a06c-3c6b218e1a80","Type":"ContainerStarted","Data":"36db16eadc5869da0a0bbbb6c72a6121409c7c782ef08affe438a1e3723d0620"}
Nov 02 01:49:40.395456 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:40.395427    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b" event={"ID":"7bef4b3e-a32d-4171-bd35-d19e9feef1e4","Type":"ContainerStarted","Data":"1408e3ce8a3aea7a7c66569514dc05028c2af17e9ffcce6b620da594484ce89a"}
Nov 02 01:49:40.395578 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:40.395471    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b" event={"ID":"7bef4b3e-a32d-4171-bd35-d19e9feef1e4","Type":"ContainerStarted","Data":"c08d83051db7a76614bfd70c7253f1080e56f6c4115544d2a036d94d514aa39a"}
Nov 02 01:49:40.434340 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:40.434307    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:40.435339 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:40.435309    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 01:49:40.435575 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:40.435553    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b" podUID="7bef4b3e-a32d-4171-bd35-d19e9feef1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 01:49:40.561408 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:40.561349    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k" podStartSLOduration=2.561327292 podStartE2EDuration="2.561327292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:40.560057322 +0000 UTC m=+1028.001093821" watchObservedRunningTime="2026-11-02 01:49:40.561327292 +0000 UTC m=+1028.002363790"
Nov 02 01:49:40.675768 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:40.675673    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:40.676041 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:40.675702    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 01:49:40.676203 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:40.676180    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k" podUID="066ee18e-0d0d-4dee-a06c-3c6b218e1a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 01:49:40.689515 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:40.689461    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b" podStartSLOduration=3.689448085 podStartE2EDuration="3.689448085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:40.687286226 +0000 UTC m=+1028.128322737" watchObservedRunningTime="2026-11-02 01:49:40.689448085 +0000 UTC m=+1028.130484582"
Nov 02 01:49:41.347355 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:41.347319    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 01:49:41.347692 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:41.347661    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz" podUID="752ab42e-3d04-4340-8481-9592789c0367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 01:49:41.389606 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:49:41.389568    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:49:41.433247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:41.432578    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 01:49:41.433247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:41.432651    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b" podUID="7bef4b3e-a32d-4171-bd35-d19e9feef1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 01:49:41.679848 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:41.679652    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 01:49:41.679848 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:41.679723    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k" podUID="066ee18e-0d0d-4dee-a06c-3c6b218e1a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 01:49:42.356248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:42.353331    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 01:49:42.356248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:42.353393    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz" podUID="752ab42e-3d04-4340-8481-9592789c0367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 01:49:42.436249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:42.435725    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 01:49:42.436249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:42.435877    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b" podUID="7bef4b3e-a32d-4171-bd35-d19e9feef1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 01:49:42.673751 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:42.673653    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 01:49:42.673969 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:42.673941    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k" podUID="066ee18e-0d0d-4dee-a06c-3c6b218e1a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 01:49:43.446272 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:43.438595    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 01:49:43.446272 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:43.438654    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b" podUID="7bef4b3e-a32d-4171-bd35-d19e9feef1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 01:49:43.560147 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:43.559037    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:43.560400 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:43.560373    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:43.561060 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:43.561038    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 01:49:43.673308 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:43.673271    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 01:49:43.673478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:43.673330    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k" podUID="066ee18e-0d0d-4dee-a06c-3c6b218e1a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 01:49:44.439249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:44.435655    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 01:49:44.439249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:44.435722    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b" podUID="7bef4b3e-a32d-4171-bd35-d19e9feef1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 01:49:44.672943 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:44.672907    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 01:49:44.673358 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:44.672961    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k" podUID="066ee18e-0d0d-4dee-a06c-3c6b218e1a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 01:49:45.436248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:45.432630    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 01:49:45.436248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:45.432690    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b" podUID="7bef4b3e-a32d-4171-bd35-d19e9feef1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 01:49:45.673157 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:45.673122    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 01:49:45.673597 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:45.673174    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k" podUID="066ee18e-0d0d-4dee-a06c-3c6b218e1a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 01:49:46.635543 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:46.635504    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:46.636181 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:46.636151    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:46.637255 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:46.637208    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 01:49:46.676950 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:46.676922    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:46.677377 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:46.677210    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:46.677832 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:46.677814    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 01:49:57.239262 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:49:57.239207    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:50:11.362525 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:50:11.362449    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:50:22.889394 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:22.889363    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:50:37.915361 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:37.915328    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"]
Nov 02 01:50:37.919550 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:37.919533    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:37.985138 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:37.985100    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkqjq\" (UniqueName: \"kubernetes.io/projected/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-kube-api-access-bkqjq\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:37.985362 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:37.985205    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-workload-certs\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:37.985440 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:37.985352    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-envoy\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:37.985440 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:37.985387    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istiod-ca-cert\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:37.985551 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:37.985533    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-credential-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:37.985607 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:37.985592    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-podinfo\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:37.985702 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:37.985689    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-token\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:37.985751 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:37.985740    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-data\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:37.985795 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:37.985772    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-workload-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:38.028989 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.028957    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"]
Nov 02 01:50:38.087021 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.086982    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-envoy\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:38.087211 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.087030    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istiod-ca-cert\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:38.087211 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.087109    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-credential-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:38.087211 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.087169    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-podinfo\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:38.087426 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.087245    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-token\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:38.087426 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.087313    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-data\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:38.087426 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.087365    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-workload-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:38.087426 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.087401    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkqjq\" (UniqueName: \"kubernetes.io/projected/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-kube-api-access-bkqjq\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:38.087619 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.087453    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-workload-certs\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:38.087619 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.087471    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-credential-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:38.087890 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.087757    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-workload-certs\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:38.087890 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.087835    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-workload-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:38.087890 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.087851    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-data\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:38.087890 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.087867    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istiod-ca-cert\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:38.089542 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.089516    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-envoy\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:38.090018 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.090001    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-podinfo\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:38.148777 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.148743    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-token\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:38.149379 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.149355    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkqjq\" (UniqueName: \"kubernetes.io/projected/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-kube-api-access-bkqjq\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:38.236424 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.236395    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:38.733260 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.733215    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"]
Nov 02 01:50:38.737996 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:50:38.737972    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb26c061_10c7_4d0c_896c_1fbd27ed00aa.slice/crio-c0fa4753e493aebc6e5e966d0901e79a92954614455c7a2f1a03c07b31332d44 WatchSource:0}: Error finding container c0fa4753e493aebc6e5e966d0901e79a92954614455c7a2f1a03c07b31332d44: Status 404 returned error can't find the container with id c0fa4753e493aebc6e5e966d0901e79a92954614455c7a2f1a03c07b31332d44
Nov 02 01:50:38.741147 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.741081    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:50:38.741349 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.741193    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:50:38.741349 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.741260    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:50:38.922557 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.922527    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph" event={"ID":"fb26c061-10c7-4d0c-896c-1fbd27ed00aa","Type":"ContainerStarted","Data":"3e906e3ea143f21d04aa923cab25c201ce66e79724cc807bab6c10f3a876d9af"}
Nov 02 01:50:38.922981 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:38.922574    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph" event={"ID":"fb26c061-10c7-4d0c-896c-1fbd27ed00aa","Type":"ContainerStarted","Data":"c0fa4753e493aebc6e5e966d0901e79a92954614455c7a2f1a03c07b31332d44"}
Nov 02 01:50:39.095660 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:39.095565    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph" podStartSLOduration=2.095550088 podStartE2EDuration="2.095550088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:50:39.092101699 +0000 UTC m=+1086.533138207" watchObservedRunningTime="2026-11-02 01:50:39.095550088 +0000 UTC m=+1086.536586625"
Nov 02 01:50:39.237607 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:39.237570    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:39.239261 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:39.239242    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 01:50:39.239336 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:39.239286    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph" podUID="fb26c061-10c7-4d0c-896c-1fbd27ed00aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 01:50:40.237494 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:40.237413    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 01:50:40.237494 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:40.237462    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph" podUID="fb26c061-10c7-4d0c-896c-1fbd27ed00aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 01:50:41.237048 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:41.237012    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 01:50:41.237270 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:41.237067    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph" podUID="fb26c061-10c7-4d0c-896c-1fbd27ed00aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 01:50:41.345045 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:50:41.345010    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:50:42.237094 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:42.237057    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 01:50:42.237392 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:42.237109    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph" podUID="fb26c061-10c7-4d0c-896c-1fbd27ed00aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 01:50:43.236790 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:43.236756    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 01:50:43.237288 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:43.236814    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph" podUID="fb26c061-10c7-4d0c-896c-1fbd27ed00aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 01:50:44.237510 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:44.237479    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 01:50:44.237935 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:44.237531    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph" podUID="fb26c061-10c7-4d0c-896c-1fbd27ed00aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 01:50:45.237108 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:45.237066    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 01:50:45.237304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:45.237131    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph" podUID="fb26c061-10c7-4d0c-896c-1fbd27ed00aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 01:50:46.241099 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:46.241071    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:46.241589 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:46.241379    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:46.242131 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:46.242116    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 01:50:51.785429 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:50:51.785395    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:51:06.945882 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:06.945841    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"]
Nov 02 01:51:06.950344 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:06.950321    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.056898 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.056863    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"]
Nov 02 01:51:07.125199 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.125159    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-envoy\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.125397 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.125274    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-workload-certs\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.125397 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.125315    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3726ebef-c581-4736-b0a3-ae417a6e8414-istiod-ca-cert\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.125549 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.125411    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-credential-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.125549 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.125493    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzqc7\" (UniqueName: \"kubernetes.io/projected/3726ebef-c581-4736-b0a3-ae417a6e8414-kube-api-access-xzqc7\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.125634 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.125559    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-podinfo\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.125686 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.125645    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-workload-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.125686 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.125664    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-token\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.125781 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.125763    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-data\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.226561 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.226528    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzqc7\" (UniqueName: \"kubernetes.io/projected/3726ebef-c581-4736-b0a3-ae417a6e8414-kube-api-access-xzqc7\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.226756 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.226585    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-podinfo\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.226756 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.226695    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-workload-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.226756 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.226720    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-token\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.226924 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.226756    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-data\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.226924 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.226860    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-envoy\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.226924 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.226898    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-workload-certs\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.227083 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.226934    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3726ebef-c581-4736-b0a3-ae417a6e8414-istiod-ca-cert\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.227083 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.226971    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-credential-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.227193 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.227128    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-workload-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.227193 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.227152    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-data\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.227333 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.227245    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-workload-certs\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.227405 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.227364    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-credential-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.227671 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.227596    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3726ebef-c581-4736-b0a3-ae417a6e8414-istiod-ca-cert\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.229211 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.229190    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-envoy\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.229362 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.229346    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-podinfo\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.287813 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.287783    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-token\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.288009 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.287990    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzqc7\" (UniqueName: \"kubernetes.io/projected/3726ebef-c581-4736-b0a3-ae417a6e8414-kube-api-access-xzqc7\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:07.564573 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:07.564486    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:08.007295 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:08.007265    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"]
Nov 02 01:51:08.008621 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:51:08.008593    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3726ebef_c581_4736_b0a3_ae417a6e8414.slice/crio-f0781d30926f6606d69f840ebaf63339f2e1ac1ccae563492bcf8bbb3907ce9d WatchSource:0}: Error finding container f0781d30926f6606d69f840ebaf63339f2e1ac1ccae563492bcf8bbb3907ce9d: Status 404 returned error can't find the container with id f0781d30926f6606d69f840ebaf63339f2e1ac1ccae563492bcf8bbb3907ce9d
Nov 02 01:51:08.011049 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:08.010981    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:51:08.011155 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:08.011088    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:51:08.011155 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:08.011126    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:51:08.176368 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:08.176334    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7" event={"ID":"3726ebef-c581-4736-b0a3-ae417a6e8414","Type":"ContainerStarted","Data":"90a1419c7d2b6bbe194675665acef20037e9d57e1e1f3be8fee60b2d03ec4d49"}
Nov 02 01:51:08.176555 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:08.176387    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7" event={"ID":"3726ebef-c581-4736-b0a3-ae417a6e8414","Type":"ContainerStarted","Data":"f0781d30926f6606d69f840ebaf63339f2e1ac1ccae563492bcf8bbb3907ce9d"}
Nov 02 01:51:08.347927 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:08.347840    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7" podStartSLOduration=2.3478262819999998 podStartE2EDuration="2.347826282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:51:08.296147009 +0000 UTC m=+1115.737183509" watchObservedRunningTime="2026-11-02 01:51:08.347826282 +0000 UTC m=+1115.788862779"
Nov 02 01:51:08.564969 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:08.564932    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:09.566630 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:09.566596    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 02 01:51:09.567112 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:09.566657    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7" podUID="3726ebef-c581-4736-b0a3-ae417a6e8414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 02 01:51:10.564852 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:10.564816    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 02 01:51:10.565064 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:10.564888    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7" podUID="3726ebef-c581-4736-b0a3-ae417a6e8414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 02 01:51:11.342883 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:51:11.342833    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:51:11.565617 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:11.565577    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 02 01:51:11.565800 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:11.565632    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7" podUID="3726ebef-c581-4736-b0a3-ae417a6e8414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 02 01:51:12.564938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:12.564906    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 02 01:51:12.565387 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:12.564955    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7" podUID="3726ebef-c581-4736-b0a3-ae417a6e8414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 02 01:51:13.565360 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:13.565322    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 02 01:51:13.565836 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:13.565377    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7" podUID="3726ebef-c581-4736-b0a3-ae417a6e8414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 02 01:51:14.878907 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:14.878859    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:14.879759 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:14.879736    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:14.880650 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:14.880632    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 01:51:20.220349 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:20.220313    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:51:41.318431 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:51:41.318391    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:51:53.050873 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:51:53.050835    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:52:11.318096 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:52:11.318016    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:52:23.714349 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.714312    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"]
Nov 02 01:52:23.720097 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.720071    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:23.882666 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.882626    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-workload-certs\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:23.882853 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.882690    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-podinfo\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:23.882853 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.882839    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-credential-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:23.882931 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.882863    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-token\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:23.882931 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.882893    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-data\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:23.882931 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.882918    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istiod-ca-cert\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:23.883036 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.883028    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mq7m\" (UniqueName: \"kubernetes.io/projected/77d448cc-8da2-4cbc-9d7e-240e51a096c0-kube-api-access-7mq7m\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:23.883090 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.883079    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-workload-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:23.883126 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.883104    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-envoy\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:23.933716 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.933663    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"]
Nov 02 01:52:23.984522 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.984436    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-data\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:23.984841 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.984816    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istiod-ca-cert\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:23.985079 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.985052    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mq7m\" (UniqueName: \"kubernetes.io/projected/77d448cc-8da2-4cbc-9d7e-240e51a096c0-kube-api-access-7mq7m\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:23.985195 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.985136    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-workload-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:23.985195 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.985171    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-envoy\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:23.985389 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.985241    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-workload-certs\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:23.985389 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.985297    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-podinfo\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:23.985507 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.985389    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-credential-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:23.985507 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.985400    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istiod-ca-cert\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:23.985507 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.985411    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-token\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:23.985796 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.985773    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-workload-certs\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:23.985870 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.985794    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-credential-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:23.986158 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.984852    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-data\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:23.986511 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.986487    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-workload-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:23.988887 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.988861    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-envoy\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:23.990295 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:23.990275    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-podinfo\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:24.069341 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:24.069031    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mq7m\" (UniqueName: \"kubernetes.io/projected/77d448cc-8da2-4cbc-9d7e-240e51a096c0-kube-api-access-7mq7m\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:24.073970 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:24.073939    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-token\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:24.340854 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:24.340753    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:24.814000 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:24.813969    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"]
Nov 02 01:52:24.815754 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:52:24.815726    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d448cc_8da2_4cbc_9d7e_240e51a096c0.slice/crio-006eca3cfe18d7327f84160914f0a95b5404a61d89e213593da28cefe98530a5 WatchSource:0}: Error finding container 006eca3cfe18d7327f84160914f0a95b5404a61d89e213593da28cefe98530a5: Status 404 returned error can't find the container with id 006eca3cfe18d7327f84160914f0a95b5404a61d89e213593da28cefe98530a5
Nov 02 01:52:24.818481 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:24.818445    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:52:24.818587 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:24.818531    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:52:24.818587 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:24.818580    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:52:25.843308 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:25.843270    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45" event={"ID":"77d448cc-8da2-4cbc-9d7e-240e51a096c0","Type":"ContainerStarted","Data":"896a02bf2e1970079559dba3d178c18f3bdab2c34502fc0c79d37e0fe13f0be5"}
Nov 02 01:52:25.843871 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:25.843848    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45" event={"ID":"77d448cc-8da2-4cbc-9d7e-240e51a096c0","Type":"ContainerStarted","Data":"006eca3cfe18d7327f84160914f0a95b5404a61d89e213593da28cefe98530a5"}
Nov 02 01:52:26.017048 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:26.016968    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45" podStartSLOduration=3.01694509 podStartE2EDuration="3.01694509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:52:26.012658547 +0000 UTC m=+1193.453695045" watchObservedRunningTime="2026-11-02 01:52:26.01694509 +0000 UTC m=+1193.457981586"
Nov 02 01:52:26.341383 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:26.341352    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:26.342842 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:26.342815    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 01:52:26.342992 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:26.342865    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45" podUID="77d448cc-8da2-4cbc-9d7e-240e51a096c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 01:52:27.341448 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:27.341409    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 01:52:27.341833 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:27.341472    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45" podUID="77d448cc-8da2-4cbc-9d7e-240e51a096c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 01:52:28.341687 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:28.341656    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 01:52:28.342100 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:28.341708    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45" podUID="77d448cc-8da2-4cbc-9d7e-240e51a096c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 01:52:29.342033 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.342001    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 01:52:29.342455 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.342053    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45" podUID="77d448cc-8da2-4cbc-9d7e-240e51a096c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 01:52:29.619849 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.619747    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"]
Nov 02 01:52:29.628964 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.628937    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:52:29.660587 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.660556    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"]
Nov 02 01:52:29.670849 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.670817    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-home\") pod \"distinct-2-w2-c62671-kserve-5c76586965-dd5r8\" (UID: \"81073143-dea0-4ec5-844f-f58d3b0f8d33\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:52:29.671026 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.670989    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-tmp-dir\") pod \"distinct-2-w2-c62671-kserve-5c76586965-dd5r8\" (UID: \"81073143-dea0-4ec5-844f-f58d3b0f8d33\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:52:29.671084 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.671039    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-dshm\") pod \"distinct-2-w2-c62671-kserve-5c76586965-dd5r8\" (UID: \"81073143-dea0-4ec5-844f-f58d3b0f8d33\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:52:29.671135 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.671127    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-model-cache\") pod \"distinct-2-w2-c62671-kserve-5c76586965-dd5r8\" (UID: \"81073143-dea0-4ec5-844f-f58d3b0f8d33\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:52:29.671172 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.671148    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81073143-dea0-4ec5-844f-f58d3b0f8d33-tls-certs\") pod \"distinct-2-w2-c62671-kserve-5c76586965-dd5r8\" (UID: \"81073143-dea0-4ec5-844f-f58d3b0f8d33\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:52:29.671689 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.671203    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2bfm\" (UniqueName: \"kubernetes.io/projected/81073143-dea0-4ec5-844f-f58d3b0f8d33-kube-api-access-r2bfm\") pod \"distinct-2-w2-c62671-kserve-5c76586965-dd5r8\" (UID: \"81073143-dea0-4ec5-844f-f58d3b0f8d33\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:52:29.719247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.714551    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c62671\"/\"kube-root-ca.crt\""
Nov 02 01:52:29.719247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.715313    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c62671\"/\"openshift-service-ca.crt\""
Nov 02 01:52:29.719247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.715396    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c62671\"/\"default-dockercfg-z5z9w\""
Nov 02 01:52:29.730411 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.730384    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c62671\"/\"distinct-2-w2-c62671-kserve-self-signed-certs\""
Nov 02 01:52:29.782310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.779998    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-model-cache\") pod \"distinct-2-w2-c62671-kserve-5c76586965-dd5r8\" (UID: \"81073143-dea0-4ec5-844f-f58d3b0f8d33\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:52:29.782310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.780056    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81073143-dea0-4ec5-844f-f58d3b0f8d33-tls-certs\") pod \"distinct-2-w2-c62671-kserve-5c76586965-dd5r8\" (UID: \"81073143-dea0-4ec5-844f-f58d3b0f8d33\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:52:29.782310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.780146    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2bfm\" (UniqueName: \"kubernetes.io/projected/81073143-dea0-4ec5-844f-f58d3b0f8d33-kube-api-access-r2bfm\") pod \"distinct-2-w2-c62671-kserve-5c76586965-dd5r8\" (UID: \"81073143-dea0-4ec5-844f-f58d3b0f8d33\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:52:29.782310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.780650    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-home\") pod \"distinct-2-w2-c62671-kserve-5c76586965-dd5r8\" (UID: \"81073143-dea0-4ec5-844f-f58d3b0f8d33\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:52:29.782310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.780785    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-tmp-dir\") pod \"distinct-2-w2-c62671-kserve-5c76586965-dd5r8\" (UID: \"81073143-dea0-4ec5-844f-f58d3b0f8d33\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:52:29.782310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.780910    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-dshm\") pod \"distinct-2-w2-c62671-kserve-5c76586965-dd5r8\" (UID: \"81073143-dea0-4ec5-844f-f58d3b0f8d33\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:52:29.783400 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.783370    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-model-cache\") pod \"distinct-2-w2-c62671-kserve-5c76586965-dd5r8\" (UID: \"81073143-dea0-4ec5-844f-f58d3b0f8d33\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:52:29.783944 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.783913    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-home\") pod \"distinct-2-w2-c62671-kserve-5c76586965-dd5r8\" (UID: \"81073143-dea0-4ec5-844f-f58d3b0f8d33\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:52:29.784454 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.784388    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-tmp-dir\") pod \"distinct-2-w2-c62671-kserve-5c76586965-dd5r8\" (UID: \"81073143-dea0-4ec5-844f-f58d3b0f8d33\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:52:29.818524 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.818451    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81073143-dea0-4ec5-844f-f58d3b0f8d33-tls-certs\") pod \"distinct-2-w2-c62671-kserve-5c76586965-dd5r8\" (UID: \"81073143-dea0-4ec5-844f-f58d3b0f8d33\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:52:29.819427 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.819369    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-dshm\") pod \"distinct-2-w2-c62671-kserve-5c76586965-dd5r8\" (UID: \"81073143-dea0-4ec5-844f-f58d3b0f8d33\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:52:29.844370 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.844296    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2bfm\" (UniqueName: \"kubernetes.io/projected/81073143-dea0-4ec5-844f-f58d3b0f8d33-kube-api-access-r2bfm\") pod \"distinct-2-w2-c62671-kserve-5c76586965-dd5r8\" (UID: \"81073143-dea0-4ec5-844f-f58d3b0f8d33\") " pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:52:29.943686 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:29.943602    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:52:30.341527 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:30.341499    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 01:52:30.341714 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:30.341557    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45" podUID="77d448cc-8da2-4cbc-9d7e-240e51a096c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 01:52:30.477165 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:30.477140    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"]
Nov 02 01:52:30.478677 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:52:30.478653    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81073143_dea0_4ec5_844f_f58d3b0f8d33.slice/crio-5a3ffae82b93f197704068530896bf8064eb877c991d5007a0628a4b9909abcc WatchSource:0}: Error finding container 5a3ffae82b93f197704068530896bf8064eb877c991d5007a0628a4b9909abcc: Status 404 returned error can't find the container with id 5a3ffae82b93f197704068530896bf8064eb877c991d5007a0628a4b9909abcc
Nov 02 01:52:30.879426 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:30.879391    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" event={"ID":"81073143-dea0-4ec5-844f-f58d3b0f8d33","Type":"ContainerStarted","Data":"b0f63abbe87826d341bb9f37171af49bf66235fa974b6b5abbcaa73ac649fe00"}
Nov 02 01:52:30.879426 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:30.879435    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" event={"ID":"81073143-dea0-4ec5-844f-f58d3b0f8d33","Type":"ContainerStarted","Data":"5a3ffae82b93f197704068530896bf8064eb877c991d5007a0628a4b9909abcc"}
Nov 02 01:52:30.944496 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:30.944460    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:52:31.019248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:30.997696    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" podStartSLOduration=1.9976777220000002 podStartE2EDuration="1.997677722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:52:30.994462063 +0000 UTC m=+1198.435498583" watchObservedRunningTime="2026-11-02 01:52:30.997677722 +0000 UTC m=+1198.438714219"
Nov 02 01:52:31.341794 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:31.341755    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 01:52:31.341978 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:31.341807    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45" podUID="77d448cc-8da2-4cbc-9d7e-240e51a096c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 01:52:32.341507 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:32.341471    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 01:52:32.341999 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:32.341534    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45" podUID="77d448cc-8da2-4cbc-9d7e-240e51a096c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 01:52:33.345566 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:33.345538    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:33.345951 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:33.345896    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:33.346710 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:33.346690    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 01:52:39.947027 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:39.946994    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:52:40.035675 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:40.035644    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:52:40.955155 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:40.955130    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:52:41.322503 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:52:41.322400    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:52:42.059341 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:42.059308    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-796832db-r8xr4"]
Nov 02 01:52:42.108366 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:42.108337    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-796832db-r8xr4"]
Nov 02 01:52:42.695135 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:42.695107    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="635321b3-6294-40e6-bca9-d68130ba9751" path="/var/lib/kubelet/pods/635321b3-6294-40e6-bca9-d68130ba9751/volumes"
Nov 02 01:52:45.814725 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:45.814688    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:52:57.048572 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.048541    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fa78e83-8zxdb"]
Nov 02 01:52:57.097673 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.097640    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fa78e83-8zxdb"]
Nov 02 01:52:57.792903 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.792870    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"]
Nov 02 01:52:57.797184 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.797169    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:52:57.846364 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.846331    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-tmp-dir\") pod \"distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8\" (UID: \"c2378991-0555-45c0-9b55-641a08637893\") " pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:52:57.846518 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.846456    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-model-cache\") pod \"distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8\" (UID: \"c2378991-0555-45c0-9b55-641a08637893\") " pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:52:57.846518 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.846490    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jfvx\" (UniqueName: \"kubernetes.io/projected/c2378991-0555-45c0-9b55-641a08637893-kube-api-access-7jfvx\") pod \"distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8\" (UID: \"c2378991-0555-45c0-9b55-641a08637893\") " pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:52:57.846681 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.846664    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2378991-0555-45c0-9b55-641a08637893-tls-certs\") pod \"distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8\" (UID: \"c2378991-0555-45c0-9b55-641a08637893\") " pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:52:57.846730 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.846697    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-dshm\") pod \"distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8\" (UID: \"c2378991-0555-45c0-9b55-641a08637893\") " pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:52:57.846795 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.846773    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-home\") pod \"distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8\" (UID: \"c2378991-0555-45c0-9b55-641a08637893\") " pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:52:57.900416 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.900385    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9470a3\"/\"openshift-service-ca.crt\""
Nov 02 01:52:57.900640 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.900612    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9470a3\"/\"default-dockercfg-mcj8f\""
Nov 02 01:52:57.900774 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.900705    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9470a3\"/\"distinct-2-w0-9470a3-kserve-self-signed-certs\""
Nov 02 01:52:57.900774 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.900735    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9470a3\"/\"kube-root-ca.crt\""
Nov 02 01:52:57.903667 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.903639    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"]
Nov 02 01:52:57.963722 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.949775    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-model-cache\") pod \"distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8\" (UID: \"c2378991-0555-45c0-9b55-641a08637893\") " pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:52:57.964050 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.964025    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jfvx\" (UniqueName: \"kubernetes.io/projected/c2378991-0555-45c0-9b55-641a08637893-kube-api-access-7jfvx\") pod \"distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8\" (UID: \"c2378991-0555-45c0-9b55-641a08637893\") " pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:52:57.967929 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.967895    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2378991-0555-45c0-9b55-641a08637893-tls-certs\") pod \"distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8\" (UID: \"c2378991-0555-45c0-9b55-641a08637893\") " pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:52:57.968668 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.968644    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-dshm\") pod \"distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8\" (UID: \"c2378991-0555-45c0-9b55-641a08637893\") " pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:52:57.972354 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.972330    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-home\") pod \"distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8\" (UID: \"c2378991-0555-45c0-9b55-641a08637893\") " pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:52:57.980058 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.980008    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-tmp-dir\") pod \"distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8\" (UID: \"c2378991-0555-45c0-9b55-641a08637893\") " pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:52:57.980567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.961885    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-model-cache\") pod \"distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8\" (UID: \"c2378991-0555-45c0-9b55-641a08637893\") " pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:52:57.980710 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.980411    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-home\") pod \"distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8\" (UID: \"c2378991-0555-45c0-9b55-641a08637893\") " pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:52:57.980807 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.980622    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-tmp-dir\") pod \"distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8\" (UID: \"c2378991-0555-45c0-9b55-641a08637893\") " pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:52:57.993514 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.993478    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-dshm\") pod \"distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8\" (UID: \"c2378991-0555-45c0-9b55-641a08637893\") " pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:52:57.998510 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:57.998421    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2378991-0555-45c0-9b55-641a08637893-tls-certs\") pod \"distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8\" (UID: \"c2378991-0555-45c0-9b55-641a08637893\") " pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:52:58.025454 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:58.025428    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jfvx\" (UniqueName: \"kubernetes.io/projected/c2378991-0555-45c0-9b55-641a08637893-kube-api-access-7jfvx\") pod \"distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8\" (UID: \"c2378991-0555-45c0-9b55-641a08637893\") " pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:52:58.109669 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:58.109602    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:52:58.489435 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:58.489406    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"]
Nov 02 01:52:58.490609 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:52:58.490582    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2378991_0555_45c0_9b55_641a08637893.slice/crio-3f9cfbe6e51e8f323aa711e54660f78f0006dd4b41eaf400442ef9b513659efe WatchSource:0}: Error finding container 3f9cfbe6e51e8f323aa711e54660f78f0006dd4b41eaf400442ef9b513659efe: Status 404 returned error can't find the container with id 3f9cfbe6e51e8f323aa711e54660f78f0006dd4b41eaf400442ef9b513659efe
Nov 02 01:52:58.695099 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:58.695067    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec7b2174-8ada-4eb8-b8e0-62b80ee430ab" path="/var/lib/kubelet/pods/ec7b2174-8ada-4eb8-b8e0-62b80ee430ab/volumes"
Nov 02 01:52:59.080692 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:59.080657    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" event={"ID":"c2378991-0555-45c0-9b55-641a08637893","Type":"ContainerStarted","Data":"cfcd8a4c3954052dfbcdcb1cf0a5b47e02500d041cd974e36a15cfaa1ed356d8"}
Nov 02 01:52:59.080835 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:59.080701    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" event={"ID":"c2378991-0555-45c0-9b55-641a08637893","Type":"ContainerStarted","Data":"3f9cfbe6e51e8f323aa711e54660f78f0006dd4b41eaf400442ef9b513659efe"}
Nov 02 01:52:59.110016 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:59.109983    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:52:59.197981 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:52:59.197929    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podStartSLOduration=2.19791597 podStartE2EDuration="2.19791597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:52:59.19746185 +0000 UTC m=+1226.638498350" watchObservedRunningTime="2026-11-02 01:52:59.19791597 +0000 UTC m=+1226.638952466"
Nov 02 01:53:08.058717 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:08.058680    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66d2ec55-rrr6v"]
Nov 02 01:53:08.112541 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:08.111626    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:53:08.135247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:08.134892    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66d2ec55-rrr6v"]
Nov 02 01:53:09.047256 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:09.031618    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="661e73e9-8fde-4f05-9fd8-e5a182b5fdc6" path="/var/lib/kubelet/pods/661e73e9-8fde-4f05-9fd8-e5a182b5fdc6/volumes"
Nov 02 01:53:09.195244 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:09.166670    5875 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:53:09.318305 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:09.286021    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" podUID="6e39fc3d-d730-4df7-8d10-2376656a88d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:53:09.410275 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:09.410235    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb" podUID="c9628178-6287-4ba3-9467-93f84ae883ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:53:11.318890 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:53:11.318854    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:53:11.919366 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:11.919332    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e89db2e-tdz2w"]
Nov 02 01:53:11.924907 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:11.924891    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e89db2e-tdz2w"
Nov 02 01:53:11.956093 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:11.956066    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7447dc9-324a-4f3e-ba29-307ab9e317c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e89db2e-tdz2w\" (UID: \"f7447dc9-324a-4f3e-ba29-307ab9e317c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e89db2e-tdz2w"
Nov 02 01:53:11.956276 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:11.956139    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcjxq\" (UniqueName: \"kubernetes.io/projected/f7447dc9-324a-4f3e-ba29-307ab9e317c4-kube-api-access-mcjxq\") pod \"maas-api-revoke-sub-4e89db2e-tdz2w\" (UID: \"f7447dc9-324a-4f3e-ba29-307ab9e317c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e89db2e-tdz2w"
Nov 02 01:53:11.956389 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:11.956372    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7447dc9-324a-4f3e-ba29-307ab9e317c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e89db2e-tdz2w\" (UID: \"f7447dc9-324a-4f3e-ba29-307ab9e317c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e89db2e-tdz2w"
Nov 02 01:53:11.977623 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:11.977588    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:53:12.027128 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:12.027098    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:53:12.030416 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:12.030391    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e89db2e-tdz2w"]
Nov 02 01:53:12.057949 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:12.057913    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7447dc9-324a-4f3e-ba29-307ab9e317c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e89db2e-tdz2w\" (UID: \"f7447dc9-324a-4f3e-ba29-307ab9e317c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e89db2e-tdz2w"
Nov 02 01:53:12.058086 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:12.057986    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcjxq\" (UniqueName: \"kubernetes.io/projected/f7447dc9-324a-4f3e-ba29-307ab9e317c4-kube-api-access-mcjxq\") pod \"maas-api-revoke-sub-4e89db2e-tdz2w\" (UID: \"f7447dc9-324a-4f3e-ba29-307ab9e317c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e89db2e-tdz2w"
Nov 02 01:53:12.058148 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:12.058111    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7447dc9-324a-4f3e-ba29-307ab9e317c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e89db2e-tdz2w\" (UID: \"f7447dc9-324a-4f3e-ba29-307ab9e317c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e89db2e-tdz2w"
Nov 02 01:53:12.058792 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:12.058734    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7447dc9-324a-4f3e-ba29-307ab9e317c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e89db2e-tdz2w\" (UID: \"f7447dc9-324a-4f3e-ba29-307ab9e317c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e89db2e-tdz2w"
Nov 02 01:53:12.060387 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:12.060367    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7447dc9-324a-4f3e-ba29-307ab9e317c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e89db2e-tdz2w\" (UID: \"f7447dc9-324a-4f3e-ba29-307ab9e317c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e89db2e-tdz2w"
Nov 02 01:53:12.119237 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:12.119201    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcjxq\" (UniqueName: \"kubernetes.io/projected/f7447dc9-324a-4f3e-ba29-307ab9e317c4-kube-api-access-mcjxq\") pod \"maas-api-revoke-sub-4e89db2e-tdz2w\" (UID: \"f7447dc9-324a-4f3e-ba29-307ab9e317c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e89db2e-tdz2w"
Nov 02 01:53:12.239604 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:12.239569    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e89db2e-tdz2w"
Nov 02 01:53:12.617305 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:12.617277    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e89db2e-tdz2w"]
Nov 02 01:53:12.618417 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:53:12.618390    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7447dc9_324a_4f3e_ba29_307ab9e317c4.slice/crio-af215e7b56343050b91649bc07feaeed25a36baba7ffcccc9057326f09b59ea9 WatchSource:0}: Error finding container af215e7b56343050b91649bc07feaeed25a36baba7ffcccc9057326f09b59ea9: Status 404 returned error can't find the container with id af215e7b56343050b91649bc07feaeed25a36baba7ffcccc9057326f09b59ea9
Nov 02 01:53:13.047753 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:13.047723    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b529ea0-6svnk"]
Nov 02 01:53:13.081198 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:13.081162    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b529ea0-6svnk"]
Nov 02 01:53:13.371292 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:13.371247    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e89db2e-tdz2w" event={"ID":"f7447dc9-324a-4f3e-ba29-307ab9e317c4","Type":"ContainerStarted","Data":"de605ed6a0569ed9bff2823edf3f9259b0087945699e355df3e2a11e1f1d9a9a"}
Nov 02 01:53:13.371471 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:13.371368    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e89db2e-tdz2w" event={"ID":"f7447dc9-324a-4f3e-ba29-307ab9e317c4","Type":"ContainerStarted","Data":"af215e7b56343050b91649bc07feaeed25a36baba7ffcccc9057326f09b59ea9"}
Nov 02 01:53:13.488258 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:13.488192    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e89db2e-tdz2w" podStartSLOduration=2.488179025 podStartE2EDuration="2.488179025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:13.485655989 +0000 UTC m=+1240.926692487" watchObservedRunningTime="2026-11-02 01:53:13.488179025 +0000 UTC m=+1240.929215523"
Nov 02 01:53:14.378557 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:14.378523    5875 generic.go:356] "Generic (PLEG): container finished" podID="f7447dc9-324a-4f3e-ba29-307ab9e317c4" containerID="de605ed6a0569ed9bff2823edf3f9259b0087945699e355df3e2a11e1f1d9a9a" exitCode=0
Nov 02 01:53:14.379025 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:14.378607    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e89db2e-tdz2w" event={"ID":"f7447dc9-324a-4f3e-ba29-307ab9e317c4","Type":"ContainerDied","Data":"de605ed6a0569ed9bff2823edf3f9259b0087945699e355df3e2a11e1f1d9a9a"}
Nov 02 01:53:14.695245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:14.695143    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b2feeac-7626-4081-a2e7-16b141a5b146" path="/var/lib/kubelet/pods/2b2feeac-7626-4081-a2e7-16b141a5b146/volumes"
Nov 02 01:53:15.535671 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:15.535645    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e89db2e-tdz2w"
Nov 02 01:53:15.610979 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:15.610939    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7447dc9-324a-4f3e-ba29-307ab9e317c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7447dc9-324a-4f3e-ba29-307ab9e317c4-maas-api-xks-ca\") pod \"f7447dc9-324a-4f3e-ba29-307ab9e317c4\" (UID: \"f7447dc9-324a-4f3e-ba29-307ab9e317c4\") "
Nov 02 01:53:15.611156 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:15.611080    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7447dc9-324a-4f3e-ba29-307ab9e317c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7447dc9-324a-4f3e-ba29-307ab9e317c4-maas-api-openshift-ca\") pod \"f7447dc9-324a-4f3e-ba29-307ab9e317c4\" (UID: \"f7447dc9-324a-4f3e-ba29-307ab9e317c4\") "
Nov 02 01:53:15.611210 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:15.611195    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7447dc9-324a-4f3e-ba29-307ab9e317c4-kube-api-access-mcjxq\" (UniqueName: \"kubernetes.io/projected/f7447dc9-324a-4f3e-ba29-307ab9e317c4-kube-api-access-mcjxq\") pod \"f7447dc9-324a-4f3e-ba29-307ab9e317c4\" (UID: \"f7447dc9-324a-4f3e-ba29-307ab9e317c4\") "
Nov 02 01:53:15.611470 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:15.611446    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7447dc9-324a-4f3e-ba29-307ab9e317c4-maas-api-openshift-ca" pod "f7447dc9-324a-4f3e-ba29-307ab9e317c4" (UID: "f7447dc9-324a-4f3e-ba29-307ab9e317c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:53:15.611708 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:15.611692    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7447dc9-324a-4f3e-ba29-307ab9e317c4-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:15.613360 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:15.613336    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7447dc9-324a-4f3e-ba29-307ab9e317c4-maas-api-xks-ca" pod "f7447dc9-324a-4f3e-ba29-307ab9e317c4" (UID: "f7447dc9-324a-4f3e-ba29-307ab9e317c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:53:15.613467 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:15.613437    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7447dc9-324a-4f3e-ba29-307ab9e317c4-kube-api-access-mcjxq" pod "f7447dc9-324a-4f3e-ba29-307ab9e317c4" (UID: "f7447dc9-324a-4f3e-ba29-307ab9e317c4"). InnerVolumeSpecName "kube-api-access-mcjxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:53:15.713129 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:15.713092    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcjxq\" (UniqueName: \"kubernetes.io/projected/f7447dc9-324a-4f3e-ba29-307ab9e317c4-kube-api-access-mcjxq\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:15.713129 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:15.713125    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7447dc9-324a-4f3e-ba29-307ab9e317c4-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:16.392878 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:16.392850    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e89db2e-tdz2w"
Nov 02 01:53:16.393051 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:16.392913    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e89db2e-tdz2w" event={"ID":"f7447dc9-324a-4f3e-ba29-307ab9e317c4","Type":"ContainerDied","Data":"af215e7b56343050b91649bc07feaeed25a36baba7ffcccc9057326f09b59ea9"}
Nov 02 01:53:16.393051 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:16.392954    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af215e7b56343050b91649bc07feaeed25a36baba7ffcccc9057326f09b59ea9"
Nov 02 01:53:16.934263 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:16.934212    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e74ca69-kvd8n"]
Nov 02 01:53:16.934924 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:16.934908    5875 cpu_manager.go:414] "containerMap: removing container" podUID="f7447dc9-324a-4f3e-ba29-307ab9e317c4" containerName="revoke-keys"
Nov 02 01:53:16.934971 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:16.934925    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7447dc9-324a-4f3e-ba29-307ab9e317c4" containerName="revoke-keys"
Nov 02 01:53:16.935037 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:16.935027    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7447dc9-324a-4f3e-ba29-307ab9e317c4" containerName="revoke-keys"
Nov 02 01:53:16.938501 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:16.938481    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e74ca69-kvd8n"
Nov 02 01:53:16.991937 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:16.991906    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:53:16.996304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:16.996280    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e74ca69-kvd8n"]
Nov 02 01:53:17.032888 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:17.032856    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11b0b310-b1b9-420f-bc26-f81d2f3f69ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e74ca69-kvd8n\" (UID: \"11b0b310-b1b9-420f-bc26-f81d2f3f69ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e74ca69-kvd8n"
Nov 02 01:53:17.033077 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:17.033054    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxpdc\" (UniqueName: \"kubernetes.io/projected/11b0b310-b1b9-420f-bc26-f81d2f3f69ef-kube-api-access-hxpdc\") pod \"maas-api-revoke-sub-2e74ca69-kvd8n\" (UID: \"11b0b310-b1b9-420f-bc26-f81d2f3f69ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e74ca69-kvd8n"
Nov 02 01:53:17.033153 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:17.033098    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11b0b310-b1b9-420f-bc26-f81d2f3f69ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e74ca69-kvd8n\" (UID: \"11b0b310-b1b9-420f-bc26-f81d2f3f69ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e74ca69-kvd8n"
Nov 02 01:53:17.041174 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:17.041146    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:53:17.134054 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:17.134019    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11b0b310-b1b9-420f-bc26-f81d2f3f69ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e74ca69-kvd8n\" (UID: \"11b0b310-b1b9-420f-bc26-f81d2f3f69ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e74ca69-kvd8n"
Nov 02 01:53:17.134252 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:17.134175    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxpdc\" (UniqueName: \"kubernetes.io/projected/11b0b310-b1b9-420f-bc26-f81d2f3f69ef-kube-api-access-hxpdc\") pod \"maas-api-revoke-sub-2e74ca69-kvd8n\" (UID: \"11b0b310-b1b9-420f-bc26-f81d2f3f69ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e74ca69-kvd8n"
Nov 02 01:53:17.134252 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:17.134210    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11b0b310-b1b9-420f-bc26-f81d2f3f69ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e74ca69-kvd8n\" (UID: \"11b0b310-b1b9-420f-bc26-f81d2f3f69ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e74ca69-kvd8n"
Nov 02 01:53:17.134853 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:17.134833    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11b0b310-b1b9-420f-bc26-f81d2f3f69ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e74ca69-kvd8n\" (UID: \"11b0b310-b1b9-420f-bc26-f81d2f3f69ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e74ca69-kvd8n"
Nov 02 01:53:17.136542 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:17.136523    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11b0b310-b1b9-420f-bc26-f81d2f3f69ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e74ca69-kvd8n\" (UID: \"11b0b310-b1b9-420f-bc26-f81d2f3f69ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e74ca69-kvd8n"
Nov 02 01:53:17.195311 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:17.195195    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxpdc\" (UniqueName: \"kubernetes.io/projected/11b0b310-b1b9-420f-bc26-f81d2f3f69ef-kube-api-access-hxpdc\") pod \"maas-api-revoke-sub-2e74ca69-kvd8n\" (UID: \"11b0b310-b1b9-420f-bc26-f81d2f3f69ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e74ca69-kvd8n"
Nov 02 01:53:17.251813 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:17.251777    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e74ca69-kvd8n"
Nov 02 01:53:17.636773 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:17.636744    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e74ca69-kvd8n"]
Nov 02 01:53:17.638484 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:53:17.638450    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b0b310_b1b9_420f_bc26_f81d2f3f69ef.slice/crio-c9c64656b306094dd16554ce039ccb5314651d78d1f0b2c6d4bb6a48e41d8277 WatchSource:0}: Error finding container c9c64656b306094dd16554ce039ccb5314651d78d1f0b2c6d4bb6a48e41d8277: Status 404 returned error can't find the container with id c9c64656b306094dd16554ce039ccb5314651d78d1f0b2c6d4bb6a48e41d8277
Nov 02 01:53:18.122076 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:18.122037    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:53:18.133382 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:18.133361    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:53:18.412629 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:18.412598    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e74ca69-kvd8n" event={"ID":"11b0b310-b1b9-420f-bc26-f81d2f3f69ef","Type":"ContainerStarted","Data":"211f6f7ff706d60b71ee2f1cbb49c2b89096fa812b3ffca05594642acf0bab72"}
Nov 02 01:53:18.412818 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:18.412658    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e74ca69-kvd8n" event={"ID":"11b0b310-b1b9-420f-bc26-f81d2f3f69ef","Type":"ContainerStarted","Data":"c9c64656b306094dd16554ce039ccb5314651d78d1f0b2c6d4bb6a48e41d8277"}
Nov 02 01:53:18.530816 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:18.530762    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e74ca69-kvd8n" podStartSLOduration=2.530748345 podStartE2EDuration="2.530748345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:18.528996557 +0000 UTC m=+1245.970033054" watchObservedRunningTime="2026-11-02 01:53:18.530748345 +0000 UTC m=+1245.971784864"
Nov 02 01:53:19.421248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:19.421183    5875 generic.go:356] "Generic (PLEG): container finished" podID="11b0b310-b1b9-420f-bc26-f81d2f3f69ef" containerID="211f6f7ff706d60b71ee2f1cbb49c2b89096fa812b3ffca05594642acf0bab72" exitCode=0
Nov 02 01:53:19.421632 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:19.421279    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e74ca69-kvd8n" event={"ID":"11b0b310-b1b9-420f-bc26-f81d2f3f69ef","Type":"ContainerDied","Data":"211f6f7ff706d60b71ee2f1cbb49c2b89096fa812b3ffca05594642acf0bab72"}
Nov 02 01:53:20.580695 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:20.580667    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e74ca69-kvd8n"
Nov 02 01:53:20.678731 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:20.678638    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11b0b310-b1b9-420f-bc26-f81d2f3f69ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11b0b310-b1b9-420f-bc26-f81d2f3f69ef-maas-api-openshift-ca\") pod \"11b0b310-b1b9-420f-bc26-f81d2f3f69ef\" (UID: \"11b0b310-b1b9-420f-bc26-f81d2f3f69ef\") "
Nov 02 01:53:20.678902 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:20.678810    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11b0b310-b1b9-420f-bc26-f81d2f3f69ef-kube-api-access-hxpdc\" (UniqueName: \"kubernetes.io/projected/11b0b310-b1b9-420f-bc26-f81d2f3f69ef-kube-api-access-hxpdc\") pod \"11b0b310-b1b9-420f-bc26-f81d2f3f69ef\" (UID: \"11b0b310-b1b9-420f-bc26-f81d2f3f69ef\") "
Nov 02 01:53:20.678994 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:20.678977    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11b0b310-b1b9-420f-bc26-f81d2f3f69ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11b0b310-b1b9-420f-bc26-f81d2f3f69ef-maas-api-xks-ca\") pod \"11b0b310-b1b9-420f-bc26-f81d2f3f69ef\" (UID: \"11b0b310-b1b9-420f-bc26-f81d2f3f69ef\") "
Nov 02 01:53:20.679059 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:20.678985    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11b0b310-b1b9-420f-bc26-f81d2f3f69ef-maas-api-openshift-ca" pod "11b0b310-b1b9-420f-bc26-f81d2f3f69ef" (UID: "11b0b310-b1b9-420f-bc26-f81d2f3f69ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:53:20.679453 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:20.679433    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11b0b310-b1b9-420f-bc26-f81d2f3f69ef-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:20.681014 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:20.680997    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11b0b310-b1b9-420f-bc26-f81d2f3f69ef-maas-api-xks-ca" pod "11b0b310-b1b9-420f-bc26-f81d2f3f69ef" (UID: "11b0b310-b1b9-420f-bc26-f81d2f3f69ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:53:20.681152 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:20.681133    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b0b310-b1b9-420f-bc26-f81d2f3f69ef-kube-api-access-hxpdc" pod "11b0b310-b1b9-420f-bc26-f81d2f3f69ef" (UID: "11b0b310-b1b9-420f-bc26-f81d2f3f69ef"). InnerVolumeSpecName "kube-api-access-hxpdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:53:20.781211 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:20.781169    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxpdc\" (UniqueName: \"kubernetes.io/projected/11b0b310-b1b9-420f-bc26-f81d2f3f69ef-kube-api-access-hxpdc\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:20.781211 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:20.781203    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11b0b310-b1b9-420f-bc26-f81d2f3f69ef-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:21.437057 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:21.437023    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e74ca69-kvd8n" event={"ID":"11b0b310-b1b9-420f-bc26-f81d2f3f69ef","Type":"ContainerDied","Data":"c9c64656b306094dd16554ce039ccb5314651d78d1f0b2c6d4bb6a48e41d8277"}
Nov 02 01:53:21.437057 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:21.437045    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e74ca69-kvd8n"
Nov 02 01:53:21.437288 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:21.437077    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9c64656b306094dd16554ce039ccb5314651d78d1f0b2c6d4bb6a48e41d8277"
Nov 02 01:53:22.926457 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:22.926426    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"]
Nov 02 01:53:22.927086 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:22.927071    5875 cpu_manager.go:414] "containerMap: removing container" podUID="11b0b310-b1b9-420f-bc26-f81d2f3f69ef" containerName="revoke-keys"
Nov 02 01:53:22.927086 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:22.927086    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11b0b310-b1b9-420f-bc26-f81d2f3f69ef" containerName="revoke-keys"
Nov 02 01:53:22.927212 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:22.927203    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="11b0b310-b1b9-420f-bc26-f81d2f3f69ef" containerName="revoke-keys"
Nov 02 01:53:22.930936 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:22.930920    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.007205 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.007171    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istiod-ca-cert\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.007379 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.007367    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-workload-certs\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.007440 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.007385    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-podinfo\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.007440 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.007421    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-data\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.007585 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.007562    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-token\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.007622 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.007596    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nthsl\" (UniqueName: \"kubernetes.io/projected/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-kube-api-access-nthsl\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.007655 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.007623    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-workload-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.007705 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.007691    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-envoy\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.007742 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.007734    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-credential-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.032641 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.032610    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"]
Nov 02 01:53:23.108555 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.108513    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-credential-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.108745 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.108572    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istiod-ca-cert\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.108745 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.108735    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-workload-certs\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.108868 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.108759    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-podinfo\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.108923 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.108881    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-credential-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.108923 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.108894    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-data\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.109028 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.108970    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-token\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.109028 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.109000    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nthsl\" (UniqueName: \"kubernetes.io/projected/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-kube-api-access-nthsl\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.109280 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.109031    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-workload-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.109280 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.109080    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-workload-certs\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.109280 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.109089    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-envoy\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.109280 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.109210    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-data\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.109715 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.109338    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istiod-ca-cert\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.109715 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.109392    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-workload-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.111491 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.111470    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-envoy\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.111580 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.111470    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-podinfo\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.169726 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.169694    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nthsl\" (UniqueName: \"kubernetes.io/projected/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-kube-api-access-nthsl\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.170013 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.169996    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-token\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.244026 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.243999    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:23.690010 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.689980    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"]
Nov 02 01:53:23.691627 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:53:23.691600    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c34e84_6fd7_43d7_9db1_7a9e755c136d.slice/crio-4c1419144f2b724bcca6fd136943c43f6ca3eabda910fd0f11bb3b3f4921e766 WatchSource:0}: Error finding container 4c1419144f2b724bcca6fd136943c43f6ca3eabda910fd0f11bb3b3f4921e766: Status 404 returned error can't find the container with id 4c1419144f2b724bcca6fd136943c43f6ca3eabda910fd0f11bb3b3f4921e766
Nov 02 01:53:23.694108 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.694072    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:53:23.694236 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.694147    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:53:23.694236 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:23.694202    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:53:24.459483 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.459441    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6" event={"ID":"a1c34e84-6fd7-43d7-9db1-7a9e755c136d","Type":"ContainerStarted","Data":"ae935c11e54ee0c3d3304bf14d1c855cbe172ab6e3b41e64d4c8bd659050eb99"}
Nov 02 01:53:24.459939 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.459518    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6" event={"ID":"a1c34e84-6fd7-43d7-9db1-7a9e755c136d","Type":"ContainerStarted","Data":"4c1419144f2b724bcca6fd136943c43f6ca3eabda910fd0f11bb3b3f4921e766"}
Nov 02 01:53:24.589955 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.589922    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"]
Nov 02 01:53:24.594234 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.594207    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.637079 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.637030    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6" podStartSLOduration=2.637016144 podStartE2EDuration="2.637016144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:24.630288554 +0000 UTC m=+1252.071325051" watchObservedRunningTime="2026-11-02 01:53:24.637016144 +0000 UTC m=+1252.078052641"
Nov 02 01:53:24.738080 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.737983    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.738319 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.738077    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm48c\" (UniqueName: \"kubernetes.io/projected/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-kube-api-access-nm48c\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.738319 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.738142    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.738319 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.738168    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.738319 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.738195    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.738319 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.738286    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.738591 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.738375    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.738591 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.738401    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.738591 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.738442    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.833390 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.833352    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"]
Nov 02 01:53:24.840024 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.839985    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.840196 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.840108    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm48c\" (UniqueName: \"kubernetes.io/projected/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-kube-api-access-nm48c\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.840196 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.840176    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.840339 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.840204    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.840339 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.840252    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.840475 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.840349    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.840533 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.840478    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.840533 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.840502    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.840639 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.840542    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.842157 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.842130    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.842437 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.842415    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.842675 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.842653    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.843560 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.843533    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.843834 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.843812    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.845713 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.845697    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.846269 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.846249    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.926281 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.926237    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm48c\" (UniqueName: \"kubernetes.io/projected/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-kube-api-access-nm48c\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.926459 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.926372    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:24.931669 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:24.931642    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:25.246883 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:25.245748    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:25.247039 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:25.246888    5875 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 01:53:25.247039 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:25.246927    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6" podUID="a1c34e84-6fd7-43d7-9db1-7a9e755c136d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 01:53:25.364446 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:25.364414    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"]
Nov 02 01:53:25.427253 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:25.425951    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:53:25.427253 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:25.426039    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:53:25.427253 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:25.426084    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:53:25.467650 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:25.467616    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k" event={"ID":"5ec9fbc9-a60c-4a82-932c-0af50b66ecab","Type":"ContainerStarted","Data":"b6bbeb20c5fbfe3870fbb45a07a045ecbe7eb238b50da5df9036b607cc3d55a5"}
Nov 02 01:53:26.245248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:26.244690    5875 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 01:53:26.245248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:26.244757    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6" podUID="a1c34e84-6fd7-43d7-9db1-7a9e755c136d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 01:53:26.476867 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:26.476833    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k" event={"ID":"5ec9fbc9-a60c-4a82-932c-0af50b66ecab","Type":"ContainerStarted","Data":"5bbac46e87a02af71b7c6653d29432628bcdf0f3f0ae71c70b984912a2bd30c5"}
Nov 02 01:53:26.725189 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:26.725118    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k" podStartSLOduration=2.725080685 podStartE2EDuration="2.725080685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:26.697015465 +0000 UTC m=+1254.138051962" watchObservedRunningTime="2026-11-02 01:53:26.725080685 +0000 UTC m=+1254.166117183"
Nov 02 01:53:26.933629 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:26.933594    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:26.934014 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:26.933989    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 02 01:53:26.934194 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:26.934171    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k" podUID="5ec9fbc9-a60c-4a82-932c-0af50b66ecab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 02 01:53:27.245668 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:27.245629    5875 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 01:53:27.246036 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:27.246007    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6" podUID="a1c34e84-6fd7-43d7-9db1-7a9e755c136d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 01:53:27.940248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:27.938330    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 02 01:53:27.940248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:27.938405    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k" podUID="5ec9fbc9-a60c-4a82-932c-0af50b66ecab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 02 01:53:28.245983 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:28.245944    5875 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 01:53:28.246404 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:28.246370    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6" podUID="a1c34e84-6fd7-43d7-9db1-7a9e755c136d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 01:53:28.941118 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:28.940654    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 02 01:53:28.941118 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:28.940719    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k" podUID="5ec9fbc9-a60c-4a82-932c-0af50b66ecab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 02 01:53:29.052328 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:29.052288    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92e7c586-zx5vs"]
Nov 02 01:53:29.055411 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:29.055389    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92e7c586-zx5vs"]
Nov 02 01:53:29.250507 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:29.250470    5875 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 01:53:29.250717 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:29.250537    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6" podUID="a1c34e84-6fd7-43d7-9db1-7a9e755c136d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 01:53:29.932581 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:29.932547    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 02 01:53:29.932764 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:29.932599    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k" podUID="5ec9fbc9-a60c-4a82-932c-0af50b66ecab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 02 01:53:30.244778 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:30.244737    5875 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 01:53:30.245176 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:30.244810    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6" podUID="a1c34e84-6fd7-43d7-9db1-7a9e755c136d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 01:53:30.696124 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:30.696036    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcd89e9b-403f-41e7-889b-615ebe4e5b24" path="/var/lib/kubelet/pods/dcd89e9b-403f-41e7-889b-615ebe4e5b24/volumes"
Nov 02 01:53:30.933311 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:30.933262    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 02 01:53:30.933511 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:30.933357    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k" podUID="5ec9fbc9-a60c-4a82-932c-0af50b66ecab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 02 01:53:31.248887 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:31.248857    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:31.249351 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:31.249123    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:31.249815 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:31.249800    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 01:53:31.932444 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:31.932411    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 02 01:53:31.932615 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:31.932486    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k" podUID="5ec9fbc9-a60c-4a82-932c-0af50b66ecab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 02 01:53:32.936577 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:32.936551    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:32.936945 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:32.936922    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:32.937767 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:32.937744    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 01:53:34.748450 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:34.748410    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"]
Nov 02 01:53:34.754092 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:34.754069    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:34.861501 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:34.861469    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"]
Nov 02 01:53:34.918743 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:34.918713    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-credential-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:34.918743 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:34.918754    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-token\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:34.919028 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:34.918902    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c95a482-2bd1-457e-9f4a-facf829b7699-istiod-ca-cert\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:34.919028 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:34.918946    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-workload-certs\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:34.919028 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:34.919014    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-data\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:34.919206 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:34.919096    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lnvd\" (UniqueName: \"kubernetes.io/projected/0c95a482-2bd1-457e-9f4a-facf829b7699-kube-api-access-7lnvd\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:34.919206 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:34.919165    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-workload-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:34.919349 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:34.919329    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-envoy\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:34.919452 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:34.919436    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-podinfo\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:35.021179 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.021091    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c95a482-2bd1-457e-9f4a-facf829b7699-istiod-ca-cert\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:35.021179 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.021125    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-workload-certs\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:35.021179 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.021166    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-data\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:35.021499 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.021201    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lnvd\" (UniqueName: \"kubernetes.io/projected/0c95a482-2bd1-457e-9f4a-facf829b7699-kube-api-access-7lnvd\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:35.021499 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.021265    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-workload-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:35.021499 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.021343    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-envoy\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:35.021499 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.021424    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-podinfo\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:35.021499 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.021460    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-credential-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:35.021499 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.021495    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-token\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:35.021823 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.021617    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-workload-certs\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:35.021823 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.021672    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-data\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:35.021823 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.021718    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-workload-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:35.021981 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.021838    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-credential-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:35.021981 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.021904    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c95a482-2bd1-457e-9f4a-facf829b7699-istiod-ca-cert\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:35.023962 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.023943    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-envoy\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:35.024158 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.024133    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-podinfo\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:35.085354 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.085319    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-token\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:35.085507 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.085427    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lnvd\" (UniqueName: \"kubernetes.io/projected/0c95a482-2bd1-457e-9f4a-facf829b7699-kube-api-access-7lnvd\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:35.257101 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.257064    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86a6787c-9xzv2"]
Nov 02 01:53:35.262550 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.262527    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a6787c-9xzv2"
Nov 02 01:53:35.314789 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.314291    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:53:35.314789 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.314511    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:53:35.318394 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.318368    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86a6787c-9xzv2"]
Nov 02 01:53:35.367164 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.367131    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:35.427237 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.427192    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-596cr\" (UniqueName: \"kubernetes.io/projected/ea66e631-971b-41b3-a868-d17e08766620-kube-api-access-596cr\") pod \"maas-api-revoke-sub-86a6787c-9xzv2\" (UID: \"ea66e631-971b-41b3-a868-d17e08766620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a6787c-9xzv2"
Nov 02 01:53:35.427439 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.427334    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea66e631-971b-41b3-a868-d17e08766620-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86a6787c-9xzv2\" (UID: \"ea66e631-971b-41b3-a868-d17e08766620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a6787c-9xzv2"
Nov 02 01:53:35.427579 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.427481    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea66e631-971b-41b3-a868-d17e08766620-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86a6787c-9xzv2\" (UID: \"ea66e631-971b-41b3-a868-d17e08766620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a6787c-9xzv2"
Nov 02 01:53:35.529332 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.529281    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-596cr\" (UniqueName: \"kubernetes.io/projected/ea66e631-971b-41b3-a868-d17e08766620-kube-api-access-596cr\") pod \"maas-api-revoke-sub-86a6787c-9xzv2\" (UID: \"ea66e631-971b-41b3-a868-d17e08766620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a6787c-9xzv2"
Nov 02 01:53:35.529536 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.529354    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea66e631-971b-41b3-a868-d17e08766620-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86a6787c-9xzv2\" (UID: \"ea66e631-971b-41b3-a868-d17e08766620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a6787c-9xzv2"
Nov 02 01:53:35.529536 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.529446    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea66e631-971b-41b3-a868-d17e08766620-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86a6787c-9xzv2\" (UID: \"ea66e631-971b-41b3-a868-d17e08766620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a6787c-9xzv2"
Nov 02 01:53:35.530033 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.530009    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea66e631-971b-41b3-a868-d17e08766620-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86a6787c-9xzv2\" (UID: \"ea66e631-971b-41b3-a868-d17e08766620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a6787c-9xzv2"
Nov 02 01:53:35.531886 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.531869    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea66e631-971b-41b3-a868-d17e08766620-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86a6787c-9xzv2\" (UID: \"ea66e631-971b-41b3-a868-d17e08766620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a6787c-9xzv2"
Nov 02 01:53:35.590058 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.589987    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-596cr\" (UniqueName: \"kubernetes.io/projected/ea66e631-971b-41b3-a868-d17e08766620-kube-api-access-596cr\") pod \"maas-api-revoke-sub-86a6787c-9xzv2\" (UID: \"ea66e631-971b-41b3-a868-d17e08766620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a6787c-9xzv2"
Nov 02 01:53:35.806660 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.806633    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"]
Nov 02 01:53:35.807826 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:53:35.807798    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c95a482_2bd1_457e_9f4a_facf829b7699.slice/crio-a8b2251a114bc3be50961c406e5b14d1ef6c2b30d14df6521638a0db962a3e05 WatchSource:0}: Error finding container a8b2251a114bc3be50961c406e5b14d1ef6c2b30d14df6521638a0db962a3e05: Status 404 returned error can't find the container with id a8b2251a114bc3be50961c406e5b14d1ef6c2b30d14df6521638a0db962a3e05
Nov 02 01:53:35.810383 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.810346    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:53:35.810503 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.810427    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:53:35.810503 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.810474    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:53:35.877186 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:35.877160    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a6787c-9xzv2"
Nov 02 01:53:36.261153 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:36.261122    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86a6787c-9xzv2"]
Nov 02 01:53:36.262460 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:53:36.262431    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea66e631_971b_41b3_a868_d17e08766620.slice/crio-49f9c15ca334dadc393eabf197dcdbb26c4be4a63735c11e79dcf3500b2775b5 WatchSource:0}: Error finding container 49f9c15ca334dadc393eabf197dcdbb26c4be4a63735c11e79dcf3500b2775b5: Status 404 returned error can't find the container with id 49f9c15ca334dadc393eabf197dcdbb26c4be4a63735c11e79dcf3500b2775b5
Nov 02 01:53:36.632292 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:36.632248    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g" event={"ID":"0c95a482-2bd1-457e-9f4a-facf829b7699","Type":"ContainerStarted","Data":"d9d62939dac7d40b7165cca55ac30e42ffa615168fcfae9e04fa1bf8c0de3dc6"}
Nov 02 01:53:36.632480 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:36.632316    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g" event={"ID":"0c95a482-2bd1-457e-9f4a-facf829b7699","Type":"ContainerStarted","Data":"a8b2251a114bc3be50961c406e5b14d1ef6c2b30d14df6521638a0db962a3e05"}
Nov 02 01:53:36.633676 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:36.633627    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a6787c-9xzv2" event={"ID":"ea66e631-971b-41b3-a868-d17e08766620","Type":"ContainerStarted","Data":"49f9c15ca334dadc393eabf197dcdbb26c4be4a63735c11e79dcf3500b2775b5"}
Nov 02 01:53:36.836194 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:36.836078    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g" podStartSLOduration=2.83606182 podStartE2EDuration="2.83606182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:36.819811021 +0000 UTC m=+1264.260847519" watchObservedRunningTime="2026-11-02 01:53:36.83606182 +0000 UTC m=+1264.277098316"
Nov 02 01:53:37.368163 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:37.368123    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:37.369639 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:37.369614    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 02 01:53:37.369748 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:37.369665    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g" podUID="0c95a482-2bd1-457e-9f4a-facf829b7699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 02 01:53:37.641612 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:37.641527    5875 generic.go:356] "Generic (PLEG): container finished" podID="ea66e631-971b-41b3-a868-d17e08766620" containerID="eea042d275d86fa5ad00c353e506254a556dd9e36acc21aa9dc73ec16dd27b66" exitCode=0
Nov 02 01:53:37.641751 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:37.641611    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a6787c-9xzv2" event={"ID":"ea66e631-971b-41b3-a868-d17e08766620","Type":"ContainerDied","Data":"eea042d275d86fa5ad00c353e506254a556dd9e36acc21aa9dc73ec16dd27b66"}
Nov 02 01:53:38.367872 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:38.367837    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 02 01:53:38.368310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:38.367886    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g" podUID="0c95a482-2bd1-457e-9f4a-facf829b7699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 02 01:53:38.837129 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:38.837100    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a6787c-9xzv2"
Nov 02 01:53:38.880251 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:38.880188    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea66e631-971b-41b3-a868-d17e08766620-kube-api-access-596cr\" (UniqueName: \"kubernetes.io/projected/ea66e631-971b-41b3-a868-d17e08766620-kube-api-access-596cr\") pod \"ea66e631-971b-41b3-a868-d17e08766620\" (UID: \"ea66e631-971b-41b3-a868-d17e08766620\") "
Nov 02 01:53:38.880499 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:38.880480    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea66e631-971b-41b3-a868-d17e08766620-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea66e631-971b-41b3-a868-d17e08766620-maas-api-xks-ca\") pod \"ea66e631-971b-41b3-a868-d17e08766620\" (UID: \"ea66e631-971b-41b3-a868-d17e08766620\") "
Nov 02 01:53:38.880574 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:38.880558    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea66e631-971b-41b3-a868-d17e08766620-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea66e631-971b-41b3-a868-d17e08766620-maas-api-openshift-ca\") pod \"ea66e631-971b-41b3-a868-d17e08766620\" (UID: \"ea66e631-971b-41b3-a868-d17e08766620\") "
Nov 02 01:53:38.880928 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:38.880902    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea66e631-971b-41b3-a868-d17e08766620-maas-api-openshift-ca" pod "ea66e631-971b-41b3-a868-d17e08766620" (UID: "ea66e631-971b-41b3-a868-d17e08766620"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:53:38.881057 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:38.881043    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea66e631-971b-41b3-a868-d17e08766620-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:38.882567 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:38.882548    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea66e631-971b-41b3-a868-d17e08766620-kube-api-access-596cr" pod "ea66e631-971b-41b3-a868-d17e08766620" (UID: "ea66e631-971b-41b3-a868-d17e08766620"). InnerVolumeSpecName "kube-api-access-596cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:53:38.882630 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:38.882564    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea66e631-971b-41b3-a868-d17e08766620-maas-api-xks-ca" pod "ea66e631-971b-41b3-a868-d17e08766620" (UID: "ea66e631-971b-41b3-a868-d17e08766620"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:53:38.982960 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:38.982923    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea66e631-971b-41b3-a868-d17e08766620-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:38.982960 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:38.982960    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-596cr\" (UniqueName: \"kubernetes.io/projected/ea66e631-971b-41b3-a868-d17e08766620-kube-api-access-596cr\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:39.368277 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:39.368183    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 02 01:53:39.368277 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:39.368259    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g" podUID="0c95a482-2bd1-457e-9f4a-facf829b7699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 02 01:53:39.656164 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:39.656088    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a6787c-9xzv2"
Nov 02 01:53:39.656164 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:39.656125    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a6787c-9xzv2" event={"ID":"ea66e631-971b-41b3-a868-d17e08766620","Type":"ContainerDied","Data":"49f9c15ca334dadc393eabf197dcdbb26c4be4a63735c11e79dcf3500b2775b5"}
Nov 02 01:53:39.656391 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:39.656176    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49f9c15ca334dadc393eabf197dcdbb26c4be4a63735c11e79dcf3500b2775b5"
Nov 02 01:53:40.169579 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:40.169494    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b6d0e87-w5t5m"]
Nov 02 01:53:40.170195 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:40.170178    5875 cpu_manager.go:414] "containerMap: removing container" podUID="ea66e631-971b-41b3-a868-d17e08766620" containerName="revoke-keys"
Nov 02 01:53:40.170260 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:40.170197    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea66e631-971b-41b3-a868-d17e08766620" containerName="revoke-keys"
Nov 02 01:53:40.170328 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:40.170317    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea66e631-971b-41b3-a868-d17e08766620" containerName="revoke-keys"
Nov 02 01:53:40.174006 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:40.173984    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6d0e87-w5t5m"
Nov 02 01:53:40.212828 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:40.212783    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b6d0e87-w5t5m\" (UID: \"19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6d0e87-w5t5m"
Nov 02 01:53:40.214598 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:40.214574    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b6d0e87-w5t5m\" (UID: \"19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6d0e87-w5t5m"
Nov 02 01:53:40.214853 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:40.214836    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8lgg\" (UniqueName: \"kubernetes.io/projected/19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6-kube-api-access-k8lgg\") pod \"maas-api-revoke-sub-6b6d0e87-w5t5m\" (UID: \"19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6d0e87-w5t5m"
Nov 02 01:53:40.227720 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:40.227469    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:53:40.227720 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:40.227628    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:53:40.231303 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:40.231266    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b6d0e87-w5t5m"]
Nov 02 01:53:40.321322 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:40.321287    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b6d0e87-w5t5m\" (UID: \"19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6d0e87-w5t5m"
Nov 02 01:53:40.321561 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:40.321541    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b6d0e87-w5t5m\" (UID: \"19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6d0e87-w5t5m"
Nov 02 01:53:40.321737 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:40.321717    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8lgg\" (UniqueName: \"kubernetes.io/projected/19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6-kube-api-access-k8lgg\") pod \"maas-api-revoke-sub-6b6d0e87-w5t5m\" (UID: \"19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6d0e87-w5t5m"
Nov 02 01:53:40.323676 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:40.323655    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b6d0e87-w5t5m\" (UID: \"19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6d0e87-w5t5m"
Nov 02 01:53:40.325607 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:40.325586    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b6d0e87-w5t5m\" (UID: \"19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6d0e87-w5t5m"
Nov 02 01:53:40.367610 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:40.367582    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 02 01:53:40.367788 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:40.367659    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g" podUID="0c95a482-2bd1-457e-9f4a-facf829b7699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 02 01:53:40.383013 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:40.382982    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8lgg\" (UniqueName: \"kubernetes.io/projected/19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6-kube-api-access-k8lgg\") pod \"maas-api-revoke-sub-6b6d0e87-w5t5m\" (UID: \"19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6d0e87-w5t5m"
Nov 02 01:53:40.488374 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:40.488340    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6d0e87-w5t5m"
Nov 02 01:53:40.894252 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:40.894205    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b6d0e87-w5t5m"]
Nov 02 01:53:40.895207 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:53:40.895178    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b0eeb4_a43a_48ac_9f2b_ff2c267c6ca6.slice/crio-24d378863f2019ec04cb2a02fd495b747086a85501926eca8eef342e792aa010 WatchSource:0}: Error finding container 24d378863f2019ec04cb2a02fd495b747086a85501926eca8eef342e792aa010: Status 404 returned error can't find the container with id 24d378863f2019ec04cb2a02fd495b747086a85501926eca8eef342e792aa010
Nov 02 01:53:41.348599 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:53:41.348568    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:53:41.368088 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:41.368058    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 02 01:53:41.368216 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:41.368106    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g" podUID="0c95a482-2bd1-457e-9f4a-facf829b7699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 02 01:53:41.675304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:41.675213    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6d0e87-w5t5m" event={"ID":"19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6","Type":"ContainerStarted","Data":"46fd194057cbcf0daae27cabb3539c86733b3ea7f0387642d922fd0df6029f34"}
Nov 02 01:53:41.675779 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:41.675758    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6d0e87-w5t5m" event={"ID":"19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6","Type":"ContainerStarted","Data":"24d378863f2019ec04cb2a02fd495b747086a85501926eca8eef342e792aa010"}
Nov 02 01:53:41.793636 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:41.793576    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6d0e87-w5t5m" podStartSLOduration=1.7935602259999999 podStartE2EDuration="1.793560226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:41.790170851 +0000 UTC m=+1269.231207350" watchObservedRunningTime="2026-11-02 01:53:41.793560226 +0000 UTC m=+1269.234596722"
Nov 02 01:53:42.368057 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:42.367974    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 02 01:53:42.368057 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:42.368026    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g" podUID="0c95a482-2bd1-457e-9f4a-facf829b7699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 02 01:53:42.685524 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:42.685429    5875 generic.go:356] "Generic (PLEG): container finished" podID="19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6" containerID="46fd194057cbcf0daae27cabb3539c86733b3ea7f0387642d922fd0df6029f34" exitCode=0
Nov 02 01:53:42.685524 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:42.685514    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6d0e87-w5t5m" event={"ID":"19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6","Type":"ContainerDied","Data":"46fd194057cbcf0daae27cabb3539c86733b3ea7f0387642d922fd0df6029f34"}
Nov 02 01:53:43.371928 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:43.371900    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:43.372360 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:43.372339    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:43.373099 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:43.373082    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 01:53:43.864617 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:43.864594    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6d0e87-w5t5m"
Nov 02 01:53:43.972010 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:43.971975    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6-kube-api-access-k8lgg\" (UniqueName: \"kubernetes.io/projected/19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6-kube-api-access-k8lgg\") pod \"19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6\" (UID: \"19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6\") "
Nov 02 01:53:43.972185 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:43.972150    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6-maas-api-openshift-ca\") pod \"19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6\" (UID: \"19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6\") "
Nov 02 01:53:43.972185 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:43.972177    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6-maas-api-xks-ca\") pod \"19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6\" (UID: \"19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6\") "
Nov 02 01:53:43.972521 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:43.972498    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6-maas-api-openshift-ca" pod "19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6" (UID: "19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:53:43.972722 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:43.972674    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:43.974214 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:43.974189    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6-kube-api-access-k8lgg" pod "19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6" (UID: "19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6"). InnerVolumeSpecName "kube-api-access-k8lgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:53:43.974359 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:43.974293    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6-maas-api-xks-ca" pod "19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6" (UID: "19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:53:44.074036 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:44.074001    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8lgg\" (UniqueName: \"kubernetes.io/projected/19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6-kube-api-access-k8lgg\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:44.074036 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:44.074030    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:44.703179 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:44.703151    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6d0e87-w5t5m"
Nov 02 01:53:44.703357 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:44.703180    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6d0e87-w5t5m" event={"ID":"19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6","Type":"ContainerDied","Data":"24d378863f2019ec04cb2a02fd495b747086a85501926eca8eef342e792aa010"}
Nov 02 01:53:44.703357 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:44.703261    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24d378863f2019ec04cb2a02fd495b747086a85501926eca8eef342e792aa010"
Nov 02 01:53:47.327514 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:47.327478    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:53:49.063451 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:49.063415    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-472402a9-7dwkg"]
Nov 02 01:53:49.065042 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:49.065014    5875 cpu_manager.go:414] "containerMap: removing container" podUID="19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6" containerName="revoke-keys"
Nov 02 01:53:49.065204 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:49.065189    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6" containerName="revoke-keys"
Nov 02 01:53:49.074266 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:49.071516    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6" containerName="revoke-keys"
Nov 02 01:53:49.077272 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:49.075949    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-472402a9-7dwkg"
Nov 02 01:53:49.235247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:49.232278    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6e8666e-3f83-4c1b-ae4c-5d76eb70069c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-472402a9-7dwkg\" (UID: \"b6e8666e-3f83-4c1b-ae4c-5d76eb70069c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472402a9-7dwkg"
Nov 02 01:53:49.235247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:49.232513    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gxsk\" (UniqueName: \"kubernetes.io/projected/b6e8666e-3f83-4c1b-ae4c-5d76eb70069c-kube-api-access-8gxsk\") pod \"maas-api-revoke-sub-472402a9-7dwkg\" (UID: \"b6e8666e-3f83-4c1b-ae4c-5d76eb70069c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472402a9-7dwkg"
Nov 02 01:53:49.235247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:49.232564    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6e8666e-3f83-4c1b-ae4c-5d76eb70069c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-472402a9-7dwkg\" (UID: \"b6e8666e-3f83-4c1b-ae4c-5d76eb70069c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472402a9-7dwkg"
Nov 02 01:53:49.247315 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:49.247121    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:53:49.247315 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:49.247265    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:53:49.347293 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:49.341660    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gxsk\" (UniqueName: \"kubernetes.io/projected/b6e8666e-3f83-4c1b-ae4c-5d76eb70069c-kube-api-access-8gxsk\") pod \"maas-api-revoke-sub-472402a9-7dwkg\" (UID: \"b6e8666e-3f83-4c1b-ae4c-5d76eb70069c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472402a9-7dwkg"
Nov 02 01:53:49.347293 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:49.341758    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6e8666e-3f83-4c1b-ae4c-5d76eb70069c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-472402a9-7dwkg\" (UID: \"b6e8666e-3f83-4c1b-ae4c-5d76eb70069c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472402a9-7dwkg"
Nov 02 01:53:49.347293 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:49.341885    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6e8666e-3f83-4c1b-ae4c-5d76eb70069c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-472402a9-7dwkg\" (UID: \"b6e8666e-3f83-4c1b-ae4c-5d76eb70069c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472402a9-7dwkg"
Nov 02 01:53:49.347293 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:49.342820    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6e8666e-3f83-4c1b-ae4c-5d76eb70069c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-472402a9-7dwkg\" (UID: \"b6e8666e-3f83-4c1b-ae4c-5d76eb70069c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472402a9-7dwkg"
Nov 02 01:53:49.532405 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:49.532134    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-472402a9-7dwkg"]
Nov 02 01:53:49.536260 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:49.532664    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6e8666e-3f83-4c1b-ae4c-5d76eb70069c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-472402a9-7dwkg\" (UID: \"b6e8666e-3f83-4c1b-ae4c-5d76eb70069c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472402a9-7dwkg"
Nov 02 01:53:49.579267 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:49.576429    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gxsk\" (UniqueName: \"kubernetes.io/projected/b6e8666e-3f83-4c1b-ae4c-5d76eb70069c-kube-api-access-8gxsk\") pod \"maas-api-revoke-sub-472402a9-7dwkg\" (UID: \"b6e8666e-3f83-4c1b-ae4c-5d76eb70069c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472402a9-7dwkg"
Nov 02 01:53:49.720788 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:49.719462    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-472402a9-7dwkg"
Nov 02 01:53:50.393554 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:50.393516    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-472402a9-7dwkg"]
Nov 02 01:53:50.395253 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:53:50.395201    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e8666e_3f83_4c1b_ae4c_5d76eb70069c.slice/crio-0302a0cd6f0d513e433f867c410817afc198e666614a7f56627d86f6bb0b091a WatchSource:0}: Error finding container 0302a0cd6f0d513e433f867c410817afc198e666614a7f56627d86f6bb0b091a: Status 404 returned error can't find the container with id 0302a0cd6f0d513e433f867c410817afc198e666614a7f56627d86f6bb0b091a
Nov 02 01:53:50.825949 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:50.825913    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-472402a9-7dwkg" event={"ID":"b6e8666e-3f83-4c1b-ae4c-5d76eb70069c","Type":"ContainerStarted","Data":"94bf7e6eeb574e4ab8d84a06e171599b275d506f03e816ea211bd48490214bfb"}
Nov 02 01:53:50.826115 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:50.825963    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-472402a9-7dwkg" event={"ID":"b6e8666e-3f83-4c1b-ae4c-5d76eb70069c","Type":"ContainerStarted","Data":"0302a0cd6f0d513e433f867c410817afc198e666614a7f56627d86f6bb0b091a"}
Nov 02 01:53:50.944447 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:50.944392    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-472402a9-7dwkg" podStartSLOduration=1.9443774249999999 podStartE2EDuration="1.944377425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:50.941118882 +0000 UTC m=+1278.382155376" watchObservedRunningTime="2026-11-02 01:53:50.944377425 +0000 UTC m=+1278.385413921"
Nov 02 01:53:51.835094 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:51.835056    5875 generic.go:356] "Generic (PLEG): container finished" podID="b6e8666e-3f83-4c1b-ae4c-5d76eb70069c" containerID="94bf7e6eeb574e4ab8d84a06e171599b275d506f03e816ea211bd48490214bfb" exitCode=0
Nov 02 01:53:51.835525 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:51.835145    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-472402a9-7dwkg" event={"ID":"b6e8666e-3f83-4c1b-ae4c-5d76eb70069c","Type":"ContainerDied","Data":"94bf7e6eeb574e4ab8d84a06e171599b275d506f03e816ea211bd48490214bfb"}
Nov 02 01:53:53.008729 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:53.008706    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-472402a9-7dwkg"
Nov 02 01:53:53.024872 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:53.024847    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6e8666e-3f83-4c1b-ae4c-5d76eb70069c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6e8666e-3f83-4c1b-ae4c-5d76eb70069c-maas-api-xks-ca\") pod \"b6e8666e-3f83-4c1b-ae4c-5d76eb70069c\" (UID: \"b6e8666e-3f83-4c1b-ae4c-5d76eb70069c\") "
Nov 02 01:53:53.024993 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:53.024909    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6e8666e-3f83-4c1b-ae4c-5d76eb70069c-kube-api-access-8gxsk\" (UniqueName: \"kubernetes.io/projected/b6e8666e-3f83-4c1b-ae4c-5d76eb70069c-kube-api-access-8gxsk\") pod \"b6e8666e-3f83-4c1b-ae4c-5d76eb70069c\" (UID: \"b6e8666e-3f83-4c1b-ae4c-5d76eb70069c\") "
Nov 02 01:53:53.025073 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:53.025019    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6e8666e-3f83-4c1b-ae4c-5d76eb70069c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6e8666e-3f83-4c1b-ae4c-5d76eb70069c-maas-api-openshift-ca\") pod \"b6e8666e-3f83-4c1b-ae4c-5d76eb70069c\" (UID: \"b6e8666e-3f83-4c1b-ae4c-5d76eb70069c\") "
Nov 02 01:53:53.025387 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:53.025361    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6e8666e-3f83-4c1b-ae4c-5d76eb70069c-maas-api-openshift-ca" pod "b6e8666e-3f83-4c1b-ae4c-5d76eb70069c" (UID: "b6e8666e-3f83-4c1b-ae4c-5d76eb70069c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:53:53.025547 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:53.025531    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6e8666e-3f83-4c1b-ae4c-5d76eb70069c-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:53.027278 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:53.027253    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6e8666e-3f83-4c1b-ae4c-5d76eb70069c-maas-api-xks-ca" pod "b6e8666e-3f83-4c1b-ae4c-5d76eb70069c" (UID: "b6e8666e-3f83-4c1b-ae4c-5d76eb70069c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:53:53.027442 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:53.027421    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e8666e-3f83-4c1b-ae4c-5d76eb70069c-kube-api-access-8gxsk" pod "b6e8666e-3f83-4c1b-ae4c-5d76eb70069c" (UID: "b6e8666e-3f83-4c1b-ae4c-5d76eb70069c"). InnerVolumeSpecName "kube-api-access-8gxsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:53:53.127594 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:53.127514    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6e8666e-3f83-4c1b-ae4c-5d76eb70069c-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:53.127594 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:53.127549    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gxsk\" (UniqueName: \"kubernetes.io/projected/b6e8666e-3f83-4c1b-ae4c-5d76eb70069c-kube-api-access-8gxsk\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:53.850164 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:53.850125    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-472402a9-7dwkg" event={"ID":"b6e8666e-3f83-4c1b-ae4c-5d76eb70069c","Type":"ContainerDied","Data":"0302a0cd6f0d513e433f867c410817afc198e666614a7f56627d86f6bb0b091a"}
Nov 02 01:53:53.850349 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:53.850190    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0302a0cd6f0d513e433f867c410817afc198e666614a7f56627d86f6bb0b091a"
Nov 02 01:53:53.850349 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:53.850143    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-472402a9-7dwkg"
Nov 02 01:53:54.977451 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:54.977405    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07650e93-llw4h"]
Nov 02 01:53:54.978040 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:54.978023    5875 cpu_manager.go:414] "containerMap: removing container" podUID="b6e8666e-3f83-4c1b-ae4c-5d76eb70069c" containerName="revoke-keys"
Nov 02 01:53:54.978040 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:54.978040    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6e8666e-3f83-4c1b-ae4c-5d76eb70069c" containerName="revoke-keys"
Nov 02 01:53:54.978194 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:54.978183    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6e8666e-3f83-4c1b-ae4c-5d76eb70069c" containerName="revoke-keys"
Nov 02 01:53:54.981900 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:54.981879    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07650e93-llw4h"
Nov 02 01:53:55.033955 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:55.033927    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:53:55.060014 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:55.059983    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx2f7\" (UniqueName: \"kubernetes.io/projected/7467eb0a-7afd-478f-94a7-95d7f22564f6-kube-api-access-bx2f7\") pod \"maas-api-revoke-sub-07650e93-llw4h\" (UID: \"7467eb0a-7afd-478f-94a7-95d7f22564f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07650e93-llw4h"
Nov 02 01:53:55.060153 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:55.060040    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7467eb0a-7afd-478f-94a7-95d7f22564f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07650e93-llw4h\" (UID: \"7467eb0a-7afd-478f-94a7-95d7f22564f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07650e93-llw4h"
Nov 02 01:53:55.060153 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:55.060103    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7467eb0a-7afd-478f-94a7-95d7f22564f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07650e93-llw4h\" (UID: \"7467eb0a-7afd-478f-94a7-95d7f22564f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07650e93-llw4h"
Nov 02 01:53:55.083552 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:55.083522    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:53:55.091536 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:55.091500    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07650e93-llw4h"]
Nov 02 01:53:55.161516 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:55.161477    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx2f7\" (UniqueName: \"kubernetes.io/projected/7467eb0a-7afd-478f-94a7-95d7f22564f6-kube-api-access-bx2f7\") pod \"maas-api-revoke-sub-07650e93-llw4h\" (UID: \"7467eb0a-7afd-478f-94a7-95d7f22564f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07650e93-llw4h"
Nov 02 01:53:55.161697 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:55.161565    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7467eb0a-7afd-478f-94a7-95d7f22564f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07650e93-llw4h\" (UID: \"7467eb0a-7afd-478f-94a7-95d7f22564f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07650e93-llw4h"
Nov 02 01:53:55.161697 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:55.161608    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7467eb0a-7afd-478f-94a7-95d7f22564f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07650e93-llw4h\" (UID: \"7467eb0a-7afd-478f-94a7-95d7f22564f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07650e93-llw4h"
Nov 02 01:53:55.162328 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:55.162298    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7467eb0a-7afd-478f-94a7-95d7f22564f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07650e93-llw4h\" (UID: \"7467eb0a-7afd-478f-94a7-95d7f22564f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07650e93-llw4h"
Nov 02 01:53:55.164161 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:55.164139    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7467eb0a-7afd-478f-94a7-95d7f22564f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07650e93-llw4h\" (UID: \"7467eb0a-7afd-478f-94a7-95d7f22564f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07650e93-llw4h"
Nov 02 01:53:55.227419 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:55.227390    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx2f7\" (UniqueName: \"kubernetes.io/projected/7467eb0a-7afd-478f-94a7-95d7f22564f6-kube-api-access-bx2f7\") pod \"maas-api-revoke-sub-07650e93-llw4h\" (UID: \"7467eb0a-7afd-478f-94a7-95d7f22564f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07650e93-llw4h"
Nov 02 01:53:55.297341 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:55.297279    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07650e93-llw4h"
Nov 02 01:53:55.674959 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:55.674935    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07650e93-llw4h"]
Nov 02 01:53:55.675909 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:53:55.675882    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7467eb0a_7afd_478f_94a7_95d7f22564f6.slice/crio-ce2fb296531356de584e8d0864fd3c1c20165702ffe99d35434d6f735cfb62cf WatchSource:0}: Error finding container ce2fb296531356de584e8d0864fd3c1c20165702ffe99d35434d6f735cfb62cf: Status 404 returned error can't find the container with id ce2fb296531356de584e8d0864fd3c1c20165702ffe99d35434d6f735cfb62cf
Nov 02 01:53:55.864286 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:55.864251    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07650e93-llw4h" event={"ID":"7467eb0a-7afd-478f-94a7-95d7f22564f6","Type":"ContainerStarted","Data":"ce2fb296531356de584e8d0864fd3c1c20165702ffe99d35434d6f735cfb62cf"}
Nov 02 01:53:56.872838 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:56.872757    5875 generic.go:356] "Generic (PLEG): container finished" podID="7467eb0a-7afd-478f-94a7-95d7f22564f6" containerID="64fd937e8dc09ac685228c3c26dd945b29540dd8a512c86d724a3d494f889cfc" exitCode=0
Nov 02 01:53:56.873204 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:56.872861    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07650e93-llw4h" event={"ID":"7467eb0a-7afd-478f-94a7-95d7f22564f6","Type":"ContainerDied","Data":"64fd937e8dc09ac685228c3c26dd945b29540dd8a512c86d724a3d494f889cfc"}
Nov 02 01:53:58.030339 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.030317    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07650e93-llw4h"
Nov 02 01:53:58.100754 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.100717    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7467eb0a-7afd-478f-94a7-95d7f22564f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7467eb0a-7afd-478f-94a7-95d7f22564f6-maas-api-openshift-ca\") pod \"7467eb0a-7afd-478f-94a7-95d7f22564f6\" (UID: \"7467eb0a-7afd-478f-94a7-95d7f22564f6\") "
Nov 02 01:53:58.100973 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.100951    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7467eb0a-7afd-478f-94a7-95d7f22564f6-kube-api-access-bx2f7\" (UniqueName: \"kubernetes.io/projected/7467eb0a-7afd-478f-94a7-95d7f22564f6-kube-api-access-bx2f7\") pod \"7467eb0a-7afd-478f-94a7-95d7f22564f6\" (UID: \"7467eb0a-7afd-478f-94a7-95d7f22564f6\") "
Nov 02 01:53:58.101050 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.100999    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7467eb0a-7afd-478f-94a7-95d7f22564f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7467eb0a-7afd-478f-94a7-95d7f22564f6-maas-api-xks-ca\") pod \"7467eb0a-7afd-478f-94a7-95d7f22564f6\" (UID: \"7467eb0a-7afd-478f-94a7-95d7f22564f6\") "
Nov 02 01:53:58.101111 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.101091    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7467eb0a-7afd-478f-94a7-95d7f22564f6-maas-api-openshift-ca" pod "7467eb0a-7afd-478f-94a7-95d7f22564f6" (UID: "7467eb0a-7afd-478f-94a7-95d7f22564f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:53:58.101692 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.101667    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7467eb0a-7afd-478f-94a7-95d7f22564f6-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:58.103420 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.103395    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7467eb0a-7afd-478f-94a7-95d7f22564f6-kube-api-access-bx2f7" pod "7467eb0a-7afd-478f-94a7-95d7f22564f6" (UID: "7467eb0a-7afd-478f-94a7-95d7f22564f6"). InnerVolumeSpecName "kube-api-access-bx2f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:53:58.103571 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.103552    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7467eb0a-7afd-478f-94a7-95d7f22564f6-maas-api-xks-ca" pod "7467eb0a-7afd-478f-94a7-95d7f22564f6" (UID: "7467eb0a-7afd-478f-94a7-95d7f22564f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:53:58.203145 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.203060    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx2f7\" (UniqueName: \"kubernetes.io/projected/7467eb0a-7afd-478f-94a7-95d7f22564f6-kube-api-access-bx2f7\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:58.203145 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.203092    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7467eb0a-7afd-478f-94a7-95d7f22564f6-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:53:58.512151 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.512118    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99e23104-7b8vf"]
Nov 02 01:53:58.512797 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.512781    5875 cpu_manager.go:414] "containerMap: removing container" podUID="7467eb0a-7afd-478f-94a7-95d7f22564f6" containerName="revoke-keys"
Nov 02 01:53:58.512845 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.512799    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7467eb0a-7afd-478f-94a7-95d7f22564f6" containerName="revoke-keys"
Nov 02 01:53:58.512940 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.512931    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="7467eb0a-7afd-478f-94a7-95d7f22564f6" containerName="revoke-keys"
Nov 02 01:53:58.516613 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.516598    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e23104-7b8vf"
Nov 02 01:53:58.574275 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.574243    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99e23104-7b8vf"]
Nov 02 01:53:58.608587 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.608557    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtfj8\" (UniqueName: \"kubernetes.io/projected/ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a-kube-api-access-rtfj8\") pod \"maas-api-revoke-sub-99e23104-7b8vf\" (UID: \"ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e23104-7b8vf"
Nov 02 01:53:58.608768 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.608604    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99e23104-7b8vf\" (UID: \"ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e23104-7b8vf"
Nov 02 01:53:58.608768 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.608717    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99e23104-7b8vf\" (UID: \"ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e23104-7b8vf"
Nov 02 01:53:58.709555 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.709528    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtfj8\" (UniqueName: \"kubernetes.io/projected/ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a-kube-api-access-rtfj8\") pod \"maas-api-revoke-sub-99e23104-7b8vf\" (UID: \"ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e23104-7b8vf"
Nov 02 01:53:58.709711 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.709574    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99e23104-7b8vf\" (UID: \"ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e23104-7b8vf"
Nov 02 01:53:58.709711 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.709636    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99e23104-7b8vf\" (UID: \"ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e23104-7b8vf"
Nov 02 01:53:58.710393 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.710371    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99e23104-7b8vf\" (UID: \"ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e23104-7b8vf"
Nov 02 01:53:58.711985 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.711968    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99e23104-7b8vf\" (UID: \"ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e23104-7b8vf"
Nov 02 01:53:58.770830 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.770762    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtfj8\" (UniqueName: \"kubernetes.io/projected/ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a-kube-api-access-rtfj8\") pod \"maas-api-revoke-sub-99e23104-7b8vf\" (UID: \"ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e23104-7b8vf"
Nov 02 01:53:58.830070 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.830039    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e23104-7b8vf"
Nov 02 01:53:58.967761 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.959008    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07650e93-llw4h" event={"ID":"7467eb0a-7afd-478f-94a7-95d7f22564f6","Type":"ContainerDied","Data":"ce2fb296531356de584e8d0864fd3c1c20165702ffe99d35434d6f735cfb62cf"}
Nov 02 01:53:58.967761 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.959067    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce2fb296531356de584e8d0864fd3c1c20165702ffe99d35434d6f735cfb62cf"
Nov 02 01:53:58.967761 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:58.959135    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07650e93-llw4h"
Nov 02 01:53:59.287427 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:59.287398    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99e23104-7b8vf"]
Nov 02 01:53:59.288851 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:53:59.288821    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee71483c_13b2_4ac9_9b3c_8f8bf8b6024a.slice/crio-33d6b896ef3c7f09643996c94fd8e1ba9f6c78baab19a2e922ee9c5033a2282a WatchSource:0}: Error finding container 33d6b896ef3c7f09643996c94fd8e1ba9f6c78baab19a2e922ee9c5033a2282a: Status 404 returned error can't find the container with id 33d6b896ef3c7f09643996c94fd8e1ba9f6c78baab19a2e922ee9c5033a2282a
Nov 02 01:53:59.968799 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:59.968762    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e23104-7b8vf" event={"ID":"ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a","Type":"ContainerStarted","Data":"8c864f6ca5a4d0ca424c7e94e9f523ed22298ed8c4ab143d4d0a8bb9419120d1"}
Nov 02 01:53:59.969059 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:53:59.968824    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e23104-7b8vf" event={"ID":"ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a","Type":"ContainerStarted","Data":"33d6b896ef3c7f09643996c94fd8e1ba9f6c78baab19a2e922ee9c5033a2282a"}
Nov 02 01:54:00.085411 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:00.085352    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e23104-7b8vf" podStartSLOduration=2.085335172 podStartE2EDuration="2.085335172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:00.083112568 +0000 UTC m=+1287.524149066" watchObservedRunningTime="2026-11-02 01:54:00.085335172 +0000 UTC m=+1287.526371668"
Nov 02 01:54:00.977728 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:00.977694    5875 generic.go:356] "Generic (PLEG): container finished" podID="ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a" containerID="8c864f6ca5a4d0ca424c7e94e9f523ed22298ed8c4ab143d4d0a8bb9419120d1" exitCode=0
Nov 02 01:54:00.978169 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:00.977738    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e23104-7b8vf" event={"ID":"ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a","Type":"ContainerDied","Data":"8c864f6ca5a4d0ca424c7e94e9f523ed22298ed8c4ab143d4d0a8bb9419120d1"}
Nov 02 01:54:01.082391 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:01.082361    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5892afd2-pnwph"]
Nov 02 01:54:01.085304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:01.085279    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5892afd2-pnwph"]
Nov 02 01:54:02.136209 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:02.136188    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e23104-7b8vf"
Nov 02 01:54:02.258287 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:02.258189    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a-kube-api-access-rtfj8\" (UniqueName: \"kubernetes.io/projected/ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a-kube-api-access-rtfj8\") pod \"ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a\" (UID: \"ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a\") "
Nov 02 01:54:02.258287 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:02.258261    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a-maas-api-xks-ca\") pod \"ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a\" (UID: \"ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a\") "
Nov 02 01:54:02.258519 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:02.258304    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a-maas-api-openshift-ca\") pod \"ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a\" (UID: \"ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a\") "
Nov 02 01:54:02.258697 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:02.258673    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a-maas-api-openshift-ca" pod "ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a" (UID: "ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:02.259002 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:02.258984    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:02.260385 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:02.260361    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a-maas-api-xks-ca" pod "ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a" (UID: "ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:02.260494 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:02.260470    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a-kube-api-access-rtfj8" pod "ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a" (UID: "ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a"). InnerVolumeSpecName "kube-api-access-rtfj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:02.360912 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:02.360873    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtfj8\" (UniqueName: \"kubernetes.io/projected/ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a-kube-api-access-rtfj8\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:02.360912 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:02.360902    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:02.694896 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:02.694822    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="323f5225-14c7-48a2-bf86-097cbb97afb9" path="/var/lib/kubelet/pods/323f5225-14c7-48a2-bf86-097cbb97afb9/volumes"
Nov 02 01:54:02.997644 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:02.997606    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e23104-7b8vf" event={"ID":"ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a","Type":"ContainerDied","Data":"33d6b896ef3c7f09643996c94fd8e1ba9f6c78baab19a2e922ee9c5033a2282a"}
Nov 02 01:54:02.997970 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:02.997950    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33d6b896ef3c7f09643996c94fd8e1ba9f6c78baab19a2e922ee9c5033a2282a"
Nov 02 01:54:02.998260 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:02.998244    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99e23104-7b8vf"
Nov 02 01:54:07.728903 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:07.728858    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5328a392-lz8ws"]
Nov 02 01:54:07.729581 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:07.729564    5875 cpu_manager.go:414] "containerMap: removing container" podUID="ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a" containerName="revoke-keys"
Nov 02 01:54:07.729581 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:07.729581    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a" containerName="revoke-keys"
Nov 02 01:54:07.729713 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:07.729697    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a" containerName="revoke-keys"
Nov 02 01:54:07.734832 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:07.734802    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a392-lz8ws"
Nov 02 01:54:07.790332 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:07.790291    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5328a392-lz8ws"]
Nov 02 01:54:07.832675 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:07.832644    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f449c9d-4431-40de-9f2e-4299fa39cb37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5328a392-lz8ws\" (UID: \"4f449c9d-4431-40de-9f2e-4299fa39cb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a392-lz8ws"
Nov 02 01:54:07.832881 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:07.832856    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f449c9d-4431-40de-9f2e-4299fa39cb37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5328a392-lz8ws\" (UID: \"4f449c9d-4431-40de-9f2e-4299fa39cb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a392-lz8ws"
Nov 02 01:54:07.832969 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:07.832956    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7kn7\" (UniqueName: \"kubernetes.io/projected/4f449c9d-4431-40de-9f2e-4299fa39cb37-kube-api-access-g7kn7\") pod \"maas-api-revoke-sub-5328a392-lz8ws\" (UID: \"4f449c9d-4431-40de-9f2e-4299fa39cb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a392-lz8ws"
Nov 02 01:54:07.834900 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:07.834878    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:54:07.835040 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:07.834978    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:54:07.933928 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:07.933895    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f449c9d-4431-40de-9f2e-4299fa39cb37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5328a392-lz8ws\" (UID: \"4f449c9d-4431-40de-9f2e-4299fa39cb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a392-lz8ws"
Nov 02 01:54:07.934092 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:07.934041    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f449c9d-4431-40de-9f2e-4299fa39cb37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5328a392-lz8ws\" (UID: \"4f449c9d-4431-40de-9f2e-4299fa39cb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a392-lz8ws"
Nov 02 01:54:07.934139 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:07.934095    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7kn7\" (UniqueName: \"kubernetes.io/projected/4f449c9d-4431-40de-9f2e-4299fa39cb37-kube-api-access-g7kn7\") pod \"maas-api-revoke-sub-5328a392-lz8ws\" (UID: \"4f449c9d-4431-40de-9f2e-4299fa39cb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a392-lz8ws"
Nov 02 01:54:07.934593 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:07.934566    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f449c9d-4431-40de-9f2e-4299fa39cb37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5328a392-lz8ws\" (UID: \"4f449c9d-4431-40de-9f2e-4299fa39cb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a392-lz8ws"
Nov 02 01:54:07.936484 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:07.936469    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f449c9d-4431-40de-9f2e-4299fa39cb37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5328a392-lz8ws\" (UID: \"4f449c9d-4431-40de-9f2e-4299fa39cb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a392-lz8ws"
Nov 02 01:54:07.996092 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:07.996015    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7kn7\" (UniqueName: \"kubernetes.io/projected/4f449c9d-4431-40de-9f2e-4299fa39cb37-kube-api-access-g7kn7\") pod \"maas-api-revoke-sub-5328a392-lz8ws\" (UID: \"4f449c9d-4431-40de-9f2e-4299fa39cb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a392-lz8ws"
Nov 02 01:54:08.049086 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:08.049060    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a392-lz8ws"
Nov 02 01:54:08.427595 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:08.427566    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5328a392-lz8ws"]
Nov 02 01:54:08.429370 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:54:08.429336    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f449c9d_4431_40de_9f2e_4299fa39cb37.slice/crio-5a57d8387c913097f1ddb3825e3e8f6052219b042a5fe761b000399801dd8659 WatchSource:0}: Error finding container 5a57d8387c913097f1ddb3825e3e8f6052219b042a5fe761b000399801dd8659: Status 404 returned error can't find the container with id 5a57d8387c913097f1ddb3825e3e8f6052219b042a5fe761b000399801dd8659
Nov 02 01:54:09.039292 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.039253    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a392-lz8ws" event={"ID":"4f449c9d-4431-40de-9f2e-4299fa39cb37","Type":"ContainerStarted","Data":"252e74770290cf0d7903746086c7c070dedf48d6f53ad4742fcf8a946ac7f75b"}
Nov 02 01:54:09.039675 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.039309    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a392-lz8ws" event={"ID":"4f449c9d-4431-40de-9f2e-4299fa39cb37","Type":"ContainerStarted","Data":"5a57d8387c913097f1ddb3825e3e8f6052219b042a5fe761b000399801dd8659"}
Nov 02 01:54:09.157978 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.157881    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a392-lz8ws" podStartSLOduration=2.157866666 podStartE2EDuration="2.157866666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:09.155315103 +0000 UTC m=+1296.596351601" watchObservedRunningTime="2026-11-02 01:54:09.157866666 +0000 UTC m=+1296.598903162"
Nov 02 01:54:09.525312 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.525275    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"]
Nov 02 01:54:09.529662 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.529639    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.661851 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.661810    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-credential-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.662046 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.661950    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-istio-envoy\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.662046 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.661979    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be86e257-8a07-418f-a12a-339e09651c93-istiod-ca-cert\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.662046 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.662010    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-workload-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.662297 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.662114    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-istio-data\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.662297 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.662133    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be86e257-8a07-418f-a12a-339e09651c93-istio-podinfo\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.662297 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.662277    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmb4j\" (UniqueName: \"kubernetes.io/projected/be86e257-8a07-418f-a12a-339e09651c93-kube-api-access-kmb4j\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.662470 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.662379    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be86e257-8a07-418f-a12a-339e09651c93-istio-token\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.662470 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.662407    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-workload-certs\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.740980 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.740947    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"]
Nov 02 01:54:09.763537 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.763504    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-istio-envoy\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.763537 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.763537    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be86e257-8a07-418f-a12a-339e09651c93-istiod-ca-cert\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.763778 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.763574    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-workload-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.763778 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.763626    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-istio-data\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.763778 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.763641    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be86e257-8a07-418f-a12a-339e09651c93-istio-podinfo\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.763778 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.763704    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmb4j\" (UniqueName: \"kubernetes.io/projected/be86e257-8a07-418f-a12a-339e09651c93-kube-api-access-kmb4j\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.763987 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.763809    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be86e257-8a07-418f-a12a-339e09651c93-istio-token\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.763987 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.763837    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-workload-certs\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.763987 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.763912    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-credential-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.764149 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.764035    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-workload-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.764149 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.764058    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-istio-data\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.764333 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.764237    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-credential-socket\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.764397 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.764371    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-workload-certs\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.764482 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.764462    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be86e257-8a07-418f-a12a-339e09651c93-istiod-ca-cert\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.766457 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.766426    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-istio-envoy\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.766768 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.766746    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be86e257-8a07-418f-a12a-339e09651c93-istio-podinfo\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.825022 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.824951    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be86e257-8a07-418f-a12a-339e09651c93-istio-token\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.825266 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.825215    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmb4j\" (UniqueName: \"kubernetes.io/projected/be86e257-8a07-418f-a12a-339e09651c93-kube-api-access-kmb4j\") pod \"e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") " pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:09.844073 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:09.844043    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:10.047639 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.047606    5875 generic.go:356] "Generic (PLEG): container finished" podID="4f449c9d-4431-40de-9f2e-4299fa39cb37" containerID="252e74770290cf0d7903746086c7c070dedf48d6f53ad4742fcf8a946ac7f75b" exitCode=0
Nov 02 01:54:10.048023 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.047684    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a392-lz8ws" event={"ID":"4f449c9d-4431-40de-9f2e-4299fa39cb37","Type":"ContainerDied","Data":"252e74770290cf0d7903746086c7c070dedf48d6f53ad4742fcf8a946ac7f75b"}
Nov 02 01:54:10.218207 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.218175    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"]
Nov 02 01:54:10.223874 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.223850    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.267651 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.267615    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"]
Nov 02 01:54:10.327821 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.327786    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"]
Nov 02 01:54:10.362323 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:54:10.362292    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe86e257_8a07_418f_a12a_339e09651c93.slice/crio-5cd645a44a6aa7bf64b15b360ca0ee9046d8d9c3cd48b130eb16b0fa7ba058a2 WatchSource:0}: Error finding container 5cd645a44a6aa7bf64b15b360ca0ee9046d8d9c3cd48b130eb16b0fa7ba058a2: Status 404 returned error can't find the container with id 5cd645a44a6aa7bf64b15b360ca0ee9046d8d9c3cd48b130eb16b0fa7ba058a2
Nov 02 01:54:10.369865 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.369821    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:54:10.370008 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.369901    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:54:10.370008 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.369935    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:54:10.377576 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.377549    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-envoy\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.377679 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.377589    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-694hm\" (UniqueName: \"kubernetes.io/projected/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-kube-api-access-694hm\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.377762 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.377746    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-podinfo\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.377812 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.377784    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istiod-ca-cert\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.377954 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.377937    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-credential-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.378070 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.378055    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-workload-certs\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.378128 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.378085    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-workload-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.378249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.378193    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-data\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.378313 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.378246    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-token\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.480161 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.480083    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-credential-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.480321 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.480305    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-workload-certs\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.480366 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.480339    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-workload-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.480401 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.480378    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-data\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.480432 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.480406    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-token\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.480510 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.480495    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-envoy\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.480546 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.480522    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-694hm\" (UniqueName: \"kubernetes.io/projected/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-kube-api-access-694hm\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.480637 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.480623    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-podinfo\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.480685 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.480663    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istiod-ca-cert\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.481570 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.481502    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istiod-ca-cert\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.481804 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.481785    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-credential-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.482007 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.481991    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-workload-certs\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.482209 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.482190    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-workload-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.482438 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.482419    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-data\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.485824 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.485777    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-envoy\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.485824 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.485805    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-podinfo\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.569273 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.555061    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-694hm\" (UniqueName: \"kubernetes.io/projected/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-kube-api-access-694hm\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.569273 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.556741    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-token\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:10.840649 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:10.840575    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:11.057253 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:11.057203    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj" event={"ID":"be86e257-8a07-418f-a12a-339e09651c93","Type":"ContainerStarted","Data":"3a8c2fd55e1d3b8c9b92e4c787133742599a5d43de66e20ab3e5c1c73d532d1c"}
Nov 02 01:54:11.057716 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:11.057274    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj" event={"ID":"be86e257-8a07-418f-a12a-339e09651c93","Type":"ContainerStarted","Data":"5cd645a44a6aa7bf64b15b360ca0ee9046d8d9c3cd48b130eb16b0fa7ba058a2"}
Nov 02 01:54:11.248937 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:11.248889    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj" podStartSLOduration=2.248875707 podStartE2EDuration="2.248875707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:11.245319999 +0000 UTC m=+1298.686356496" watchObservedRunningTime="2026-11-02 01:54:11.248875707 +0000 UTC m=+1298.689912204"
Nov 02 01:54:11.265208 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:11.265181    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a392-lz8ws"
Nov 02 01:54:11.323355 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:11.323313    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"]
Nov 02 01:54:11.330763 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:11.330729    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:54:11.330872 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:11.330800    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:54:11.330872 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:11.330830    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:54:11.357785 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:54:11.357755    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:54:11.394534 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:11.394384    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f449c9d-4431-40de-9f2e-4299fa39cb37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f449c9d-4431-40de-9f2e-4299fa39cb37-maas-api-openshift-ca\") pod \"4f449c9d-4431-40de-9f2e-4299fa39cb37\" (UID: \"4f449c9d-4431-40de-9f2e-4299fa39cb37\") "
Nov 02 01:54:11.394683 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:11.394658    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f449c9d-4431-40de-9f2e-4299fa39cb37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f449c9d-4431-40de-9f2e-4299fa39cb37-maas-api-xks-ca\") pod \"4f449c9d-4431-40de-9f2e-4299fa39cb37\" (UID: \"4f449c9d-4431-40de-9f2e-4299fa39cb37\") "
Nov 02 01:54:11.394777 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:11.394762    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f449c9d-4431-40de-9f2e-4299fa39cb37-kube-api-access-g7kn7\" (UniqueName: \"kubernetes.io/projected/4f449c9d-4431-40de-9f2e-4299fa39cb37-kube-api-access-g7kn7\") pod \"4f449c9d-4431-40de-9f2e-4299fa39cb37\" (UID: \"4f449c9d-4431-40de-9f2e-4299fa39cb37\") "
Nov 02 01:54:11.395675 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:11.395646    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f449c9d-4431-40de-9f2e-4299fa39cb37-maas-api-openshift-ca" pod "4f449c9d-4431-40de-9f2e-4299fa39cb37" (UID: "4f449c9d-4431-40de-9f2e-4299fa39cb37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:11.427248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:11.397913    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f449c9d-4431-40de-9f2e-4299fa39cb37-kube-api-access-g7kn7" pod "4f449c9d-4431-40de-9f2e-4299fa39cb37" (UID: "4f449c9d-4431-40de-9f2e-4299fa39cb37"). InnerVolumeSpecName "kube-api-access-g7kn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:11.427248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:11.412349    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f449c9d-4431-40de-9f2e-4299fa39cb37-maas-api-xks-ca" pod "4f449c9d-4431-40de-9f2e-4299fa39cb37" (UID: "4f449c9d-4431-40de-9f2e-4299fa39cb37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:11.496615 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:11.496589    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f449c9d-4431-40de-9f2e-4299fa39cb37-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:11.496615 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:11.496614    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7kn7\" (UniqueName: \"kubernetes.io/projected/4f449c9d-4431-40de-9f2e-4299fa39cb37-kube-api-access-g7kn7\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:11.496865 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:11.496627    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f449c9d-4431-40de-9f2e-4299fa39cb37-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:11.845067 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:11.845034    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:11.846736 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:11.846712    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 01:54:11.846797 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:11.846769    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj" podUID="be86e257-8a07-418f-a12a-339e09651c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 01:54:12.066191 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.066157    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf" event={"ID":"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1","Type":"ContainerStarted","Data":"cf821bd51f4731fc3820f3840d4a7b34742fde60166cf8fb948c9574598237f6"}
Nov 02 01:54:12.066659 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.066234    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf" event={"ID":"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1","Type":"ContainerStarted","Data":"d42131f02b2350d12ee77e942604a636f739b3ca04b6291ec44c00adbe5b6b44"}
Nov 02 01:54:12.067908 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.067884    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a392-lz8ws"
Nov 02 01:54:12.068023 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.067925    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5328a392-lz8ws" event={"ID":"4f449c9d-4431-40de-9f2e-4299fa39cb37","Type":"ContainerDied","Data":"5a57d8387c913097f1ddb3825e3e8f6052219b042a5fe761b000399801dd8659"}
Nov 02 01:54:12.068023 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.067970    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a57d8387c913097f1ddb3825e3e8f6052219b042a5fe761b000399801dd8659"
Nov 02 01:54:12.190637 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.190536    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf" podStartSLOduration=2.190522758 podStartE2EDuration="2.190522758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:12.186327521 +0000 UTC m=+1299.627364040" watchObservedRunningTime="2026-11-02 01:54:12.190522758 +0000 UTC m=+1299.631559255"
Nov 02 01:54:12.550245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.549153    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a4e411c-xhrxx"]
Nov 02 01:54:12.550464 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.550286    5875 cpu_manager.go:414] "containerMap: removing container" podUID="4f449c9d-4431-40de-9f2e-4299fa39cb37" containerName="revoke-keys"
Nov 02 01:54:12.550464 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.550312    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f449c9d-4431-40de-9f2e-4299fa39cb37" containerName="revoke-keys"
Nov 02 01:54:12.550558 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.550506    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f449c9d-4431-40de-9f2e-4299fa39cb37" containerName="revoke-keys"
Nov 02 01:54:12.555240 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.555064    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4e411c-xhrxx"
Nov 02 01:54:12.644247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.640739    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a4e411c-xhrxx"]
Nov 02 01:54:12.666460 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.666427    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:54:12.666853 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.666832    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:54:12.667716 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.667689    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f96dcece-5rkqn"]
Nov 02 01:54:12.673048 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.673029    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f96dcece-5rkqn"]
Nov 02 01:54:12.673146 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.673110    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96dcece-5rkqn"
Nov 02 01:54:12.768137 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.767915    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zck46\" (UniqueName: \"kubernetes.io/projected/e4d60ff0-bf59-4022-a94f-1c3e560a7a6c-kube-api-access-zck46\") pod \"maas-api-revoke-sub-8a4e411c-xhrxx\" (UID: \"e4d60ff0-bf59-4022-a94f-1c3e560a7a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4e411c-xhrxx"
Nov 02 01:54:12.768137 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.768104    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4d60ff0-bf59-4022-a94f-1c3e560a7a6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a4e411c-xhrxx\" (UID: \"e4d60ff0-bf59-4022-a94f-1c3e560a7a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4e411c-xhrxx"
Nov 02 01:54:12.768663 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.768634    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4d60ff0-bf59-4022-a94f-1c3e560a7a6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a4e411c-xhrxx\" (UID: \"e4d60ff0-bf59-4022-a94f-1c3e560a7a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4e411c-xhrxx"
Nov 02 01:54:12.847334 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.847177    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:12.847871 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.847834    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 01:54:12.848039 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.848014    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj" podUID="be86e257-8a07-418f-a12a-339e09651c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 01:54:12.853764 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.852469    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 02 01:54:12.853764 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.852519    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf" podUID="24b5aa29-7f6a-46fb-a73f-6de2d308a2b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 02 01:54:12.869676 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.869645    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4d60ff0-bf59-4022-a94f-1c3e560a7a6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a4e411c-xhrxx\" (UID: \"e4d60ff0-bf59-4022-a94f-1c3e560a7a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4e411c-xhrxx"
Nov 02 01:54:12.870170 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.870144    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9fe9e10-c020-4f72-a108-72cabf18cbae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f96dcece-5rkqn\" (UID: \"b9fe9e10-c020-4f72-a108-72cabf18cbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96dcece-5rkqn"
Nov 02 01:54:12.870273 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.870213    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zck46\" (UniqueName: \"kubernetes.io/projected/e4d60ff0-bf59-4022-a94f-1c3e560a7a6c-kube-api-access-zck46\") pod \"maas-api-revoke-sub-8a4e411c-xhrxx\" (UID: \"e4d60ff0-bf59-4022-a94f-1c3e560a7a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4e411c-xhrxx"
Nov 02 01:54:12.870374 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.870286    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9fe9e10-c020-4f72-a108-72cabf18cbae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f96dcece-5rkqn\" (UID: \"b9fe9e10-c020-4f72-a108-72cabf18cbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96dcece-5rkqn"
Nov 02 01:54:12.870374 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.870353    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg8st\" (UniqueName: \"kubernetes.io/projected/b9fe9e10-c020-4f72-a108-72cabf18cbae-kube-api-access-kg8st\") pod \"maas-api-revoke-sub-f96dcece-5rkqn\" (UID: \"b9fe9e10-c020-4f72-a108-72cabf18cbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96dcece-5rkqn"
Nov 02 01:54:12.870611 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.870493    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4d60ff0-bf59-4022-a94f-1c3e560a7a6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a4e411c-xhrxx\" (UID: \"e4d60ff0-bf59-4022-a94f-1c3e560a7a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4e411c-xhrxx"
Nov 02 01:54:12.871193 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.871169    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4d60ff0-bf59-4022-a94f-1c3e560a7a6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a4e411c-xhrxx\" (UID: \"e4d60ff0-bf59-4022-a94f-1c3e560a7a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4e411c-xhrxx"
Nov 02 01:54:12.872261 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.872240    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4d60ff0-bf59-4022-a94f-1c3e560a7a6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a4e411c-xhrxx\" (UID: \"e4d60ff0-bf59-4022-a94f-1c3e560a7a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4e411c-xhrxx"
Nov 02 01:54:12.950735 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.950691    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zck46\" (UniqueName: \"kubernetes.io/projected/e4d60ff0-bf59-4022-a94f-1c3e560a7a6c-kube-api-access-zck46\") pod \"maas-api-revoke-sub-8a4e411c-xhrxx\" (UID: \"e4d60ff0-bf59-4022-a94f-1c3e560a7a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4e411c-xhrxx"
Nov 02 01:54:12.971595 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.971563    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9fe9e10-c020-4f72-a108-72cabf18cbae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f96dcece-5rkqn\" (UID: \"b9fe9e10-c020-4f72-a108-72cabf18cbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96dcece-5rkqn"
Nov 02 01:54:12.971788 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.971620    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9fe9e10-c020-4f72-a108-72cabf18cbae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f96dcece-5rkqn\" (UID: \"b9fe9e10-c020-4f72-a108-72cabf18cbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96dcece-5rkqn"
Nov 02 01:54:12.971788 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.971671    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg8st\" (UniqueName: \"kubernetes.io/projected/b9fe9e10-c020-4f72-a108-72cabf18cbae-kube-api-access-kg8st\") pod \"maas-api-revoke-sub-f96dcece-5rkqn\" (UID: \"b9fe9e10-c020-4f72-a108-72cabf18cbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96dcece-5rkqn"
Nov 02 01:54:12.972585 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.972555    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9fe9e10-c020-4f72-a108-72cabf18cbae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f96dcece-5rkqn\" (UID: \"b9fe9e10-c020-4f72-a108-72cabf18cbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96dcece-5rkqn"
Nov 02 01:54:12.974351 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:12.974331    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9fe9e10-c020-4f72-a108-72cabf18cbae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f96dcece-5rkqn\" (UID: \"b9fe9e10-c020-4f72-a108-72cabf18cbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96dcece-5rkqn"
Nov 02 01:54:13.042545 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:13.034310    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg8st\" (UniqueName: \"kubernetes.io/projected/b9fe9e10-c020-4f72-a108-72cabf18cbae-kube-api-access-kg8st\") pod \"maas-api-revoke-sub-f96dcece-5rkqn\" (UID: \"b9fe9e10-c020-4f72-a108-72cabf18cbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96dcece-5rkqn"
Nov 02 01:54:13.169360 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:13.169282    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4e411c-xhrxx"
Nov 02 01:54:13.304878 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:13.304629    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96dcece-5rkqn"
Nov 02 01:54:13.669581 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:13.669082    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a4e411c-xhrxx"]
Nov 02 01:54:13.843276 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:13.841920    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 02 01:54:13.843276 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:13.841981    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf" podUID="24b5aa29-7f6a-46fb-a73f-6de2d308a2b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 02 01:54:13.845267 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:13.845209    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 01:54:13.845376 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:13.845288    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj" podUID="be86e257-8a07-418f-a12a-339e09651c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 01:54:14.042329 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:14.041118    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f96dcece-5rkqn"]
Nov 02 01:54:14.061454 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:54:14.061410    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9fe9e10_c020_4f72_a108_72cabf18cbae.slice/crio-61e392454846b833e1ae633f3ddff22ca25698a4647a4d1b9f3e4d3f4eae9f27 WatchSource:0}: Error finding container 61e392454846b833e1ae633f3ddff22ca25698a4647a4d1b9f3e4d3f4eae9f27: Status 404 returned error can't find the container with id 61e392454846b833e1ae633f3ddff22ca25698a4647a4d1b9f3e4d3f4eae9f27
Nov 02 01:54:14.085035 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:14.085004    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96dcece-5rkqn" event={"ID":"b9fe9e10-c020-4f72-a108-72cabf18cbae","Type":"ContainerStarted","Data":"61e392454846b833e1ae633f3ddff22ca25698a4647a4d1b9f3e4d3f4eae9f27"}
Nov 02 01:54:14.086357 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:14.086333    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4e411c-xhrxx" event={"ID":"e4d60ff0-bf59-4022-a94f-1c3e560a7a6c","Type":"ContainerStarted","Data":"37705f7a32b592995a9a5da3d341f3684379e9747ee17b1a2bb92b8293727490"}
Nov 02 01:54:14.842634 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:14.842595    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 02 01:54:14.843240 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:14.843191    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf" podUID="24b5aa29-7f6a-46fb-a73f-6de2d308a2b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 02 01:54:14.850419 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:14.850321    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 01:54:14.850419 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:14.850364    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj" podUID="be86e257-8a07-418f-a12a-339e09651c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 01:54:15.095296 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:15.094829    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96dcece-5rkqn" event={"ID":"b9fe9e10-c020-4f72-a108-72cabf18cbae","Type":"ContainerStarted","Data":"8c8ba4304a281e3a45fa878c945c97ffc76ca177649fd74435c4226d995b8a7b"}
Nov 02 01:54:15.096756 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:15.096727    5875 generic.go:356] "Generic (PLEG): container finished" podID="e4d60ff0-bf59-4022-a94f-1c3e560a7a6c" containerID="5a42151097f09cbb0759137ebb3074f88589148aa995216c3f33a15819b613ba" exitCode=0
Nov 02 01:54:15.096818 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:15.096795    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4e411c-xhrxx" event={"ID":"e4d60ff0-bf59-4022-a94f-1c3e560a7a6c","Type":"ContainerDied","Data":"5a42151097f09cbb0759137ebb3074f88589148aa995216c3f33a15819b613ba"}
Nov 02 01:54:15.228248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:15.227238    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96dcece-5rkqn" podStartSLOduration=3.227199697 podStartE2EDuration="3.227199697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:15.226466909 +0000 UTC m=+1302.667503407" watchObservedRunningTime="2026-11-02 01:54:15.227199697 +0000 UTC m=+1302.668236195"
Nov 02 01:54:15.848190 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:15.847528    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 01:54:15.848190 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:15.847573    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj" podUID="be86e257-8a07-418f-a12a-339e09651c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 01:54:15.848190 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:15.848041    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 02 01:54:15.848190 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:15.848069    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf" podUID="24b5aa29-7f6a-46fb-a73f-6de2d308a2b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 02 01:54:16.152931 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:16.152846    5875 generic.go:356] "Generic (PLEG): container finished" podID="b9fe9e10-c020-4f72-a108-72cabf18cbae" containerID="8c8ba4304a281e3a45fa878c945c97ffc76ca177649fd74435c4226d995b8a7b" exitCode=0
Nov 02 01:54:16.153080 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:16.152896    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96dcece-5rkqn" event={"ID":"b9fe9e10-c020-4f72-a108-72cabf18cbae","Type":"ContainerDied","Data":"8c8ba4304a281e3a45fa878c945c97ffc76ca177649fd74435c4226d995b8a7b"}
Nov 02 01:54:16.464049 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:16.463977    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4e411c-xhrxx"
Nov 02 01:54:16.487292 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:16.487262    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4d60ff0-bf59-4022-a94f-1c3e560a7a6c-kube-api-access-zck46\" (UniqueName: \"kubernetes.io/projected/e4d60ff0-bf59-4022-a94f-1c3e560a7a6c-kube-api-access-zck46\") pod \"e4d60ff0-bf59-4022-a94f-1c3e560a7a6c\" (UID: \"e4d60ff0-bf59-4022-a94f-1c3e560a7a6c\") "
Nov 02 01:54:16.487438 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:16.487409    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4d60ff0-bf59-4022-a94f-1c3e560a7a6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4d60ff0-bf59-4022-a94f-1c3e560a7a6c-maas-api-openshift-ca\") pod \"e4d60ff0-bf59-4022-a94f-1c3e560a7a6c\" (UID: \"e4d60ff0-bf59-4022-a94f-1c3e560a7a6c\") "
Nov 02 01:54:16.487551 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:16.487534    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4d60ff0-bf59-4022-a94f-1c3e560a7a6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4d60ff0-bf59-4022-a94f-1c3e560a7a6c-maas-api-xks-ca\") pod \"e4d60ff0-bf59-4022-a94f-1c3e560a7a6c\" (UID: \"e4d60ff0-bf59-4022-a94f-1c3e560a7a6c\") "
Nov 02 01:54:16.487827 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:16.487805    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4d60ff0-bf59-4022-a94f-1c3e560a7a6c-maas-api-openshift-ca" pod "e4d60ff0-bf59-4022-a94f-1c3e560a7a6c" (UID: "e4d60ff0-bf59-4022-a94f-1c3e560a7a6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:16.487992 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:16.487973    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4d60ff0-bf59-4022-a94f-1c3e560a7a6c-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:16.489523 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:16.489496    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d60ff0-bf59-4022-a94f-1c3e560a7a6c-kube-api-access-zck46" pod "e4d60ff0-bf59-4022-a94f-1c3e560a7a6c" (UID: "e4d60ff0-bf59-4022-a94f-1c3e560a7a6c"). InnerVolumeSpecName "kube-api-access-zck46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:16.489994 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:16.489976    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4d60ff0-bf59-4022-a94f-1c3e560a7a6c-maas-api-xks-ca" pod "e4d60ff0-bf59-4022-a94f-1c3e560a7a6c" (UID: "e4d60ff0-bf59-4022-a94f-1c3e560a7a6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:16.589963 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:16.589927    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4d60ff0-bf59-4022-a94f-1c3e560a7a6c-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:16.589963 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:16.589963    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zck46\" (UniqueName: \"kubernetes.io/projected/e4d60ff0-bf59-4022-a94f-1c3e560a7a6c-kube-api-access-zck46\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:16.841705 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:16.841669    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 02 01:54:16.841856 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:16.841737    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf" podUID="24b5aa29-7f6a-46fb-a73f-6de2d308a2b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 02 01:54:16.845007 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:16.844985    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 01:54:16.845081 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:16.845026    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj" podUID="be86e257-8a07-418f-a12a-339e09651c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 01:54:17.053016 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:17.052916    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74fdb954-fm82f"]
Nov 02 01:54:17.056480 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:17.056457    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74fdb954-fm82f"]
Nov 02 01:54:17.162726 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:17.162701    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4e411c-xhrxx"
Nov 02 01:54:17.162912 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:17.162735    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4e411c-xhrxx" event={"ID":"e4d60ff0-bf59-4022-a94f-1c3e560a7a6c","Type":"ContainerDied","Data":"37705f7a32b592995a9a5da3d341f3684379e9747ee17b1a2bb92b8293727490"}
Nov 02 01:54:17.162912 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:17.162789    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37705f7a32b592995a9a5da3d341f3684379e9747ee17b1a2bb92b8293727490"
Nov 02 01:54:17.372028 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:17.371999    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96dcece-5rkqn"
Nov 02 01:54:17.440086 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:17.440050    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9fe9e10-c020-4f72-a108-72cabf18cbae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9fe9e10-c020-4f72-a108-72cabf18cbae-maas-api-xks-ca\") pod \"b9fe9e10-c020-4f72-a108-72cabf18cbae\" (UID: \"b9fe9e10-c020-4f72-a108-72cabf18cbae\") "
Nov 02 01:54:17.440303 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:17.440116    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9fe9e10-c020-4f72-a108-72cabf18cbae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9fe9e10-c020-4f72-a108-72cabf18cbae-maas-api-openshift-ca\") pod \"b9fe9e10-c020-4f72-a108-72cabf18cbae\" (UID: \"b9fe9e10-c020-4f72-a108-72cabf18cbae\") "
Nov 02 01:54:17.440303 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:17.440278    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9fe9e10-c020-4f72-a108-72cabf18cbae-kube-api-access-kg8st\" (UniqueName: \"kubernetes.io/projected/b9fe9e10-c020-4f72-a108-72cabf18cbae-kube-api-access-kg8st\") pod \"b9fe9e10-c020-4f72-a108-72cabf18cbae\" (UID: \"b9fe9e10-c020-4f72-a108-72cabf18cbae\") "
Nov 02 01:54:17.440529 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:17.440508    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9fe9e10-c020-4f72-a108-72cabf18cbae-maas-api-openshift-ca" pod "b9fe9e10-c020-4f72-a108-72cabf18cbae" (UID: "b9fe9e10-c020-4f72-a108-72cabf18cbae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:17.440933 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:17.440907    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9fe9e10-c020-4f72-a108-72cabf18cbae-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:17.442659 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:17.442629    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9fe9e10-c020-4f72-a108-72cabf18cbae-kube-api-access-kg8st" pod "b9fe9e10-c020-4f72-a108-72cabf18cbae" (UID: "b9fe9e10-c020-4f72-a108-72cabf18cbae"). InnerVolumeSpecName "kube-api-access-kg8st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:17.442889 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:17.442860    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9fe9e10-c020-4f72-a108-72cabf18cbae-maas-api-xks-ca" pod "b9fe9e10-c020-4f72-a108-72cabf18cbae" (UID: "b9fe9e10-c020-4f72-a108-72cabf18cbae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:17.542520 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:17.542483    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9fe9e10-c020-4f72-a108-72cabf18cbae-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:17.542520 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:17.542524    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg8st\" (UniqueName: \"kubernetes.io/projected/b9fe9e10-c020-4f72-a108-72cabf18cbae-kube-api-access-kg8st\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:17.841401 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:17.841370    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 02 01:54:17.841584 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:17.841427    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf" podUID="24b5aa29-7f6a-46fb-a73f-6de2d308a2b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 02 01:54:18.048377 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:18.048337    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:18.049062 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:18.049045    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:18.049878 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:18.049856    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 01:54:18.180434 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:18.180345    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96dcece-5rkqn" event={"ID":"b9fe9e10-c020-4f72-a108-72cabf18cbae","Type":"ContainerDied","Data":"61e392454846b833e1ae633f3ddff22ca25698a4647a4d1b9f3e4d3f4eae9f27"}
Nov 02 01:54:18.180434 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:18.180390    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96dcece-5rkqn"
Nov 02 01:54:18.180434 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:18.180426    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61e392454846b833e1ae633f3ddff22ca25698a4647a4d1b9f3e4d3f4eae9f27"
Nov 02 01:54:18.695833 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:18.695801    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="709291d6-23c8-4d12-8285-6a2630fa1e7c" path="/var/lib/kubelet/pods/709291d6-23c8-4d12-8285-6a2630fa1e7c/volumes"
Nov 02 01:54:19.285788 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:19.285732    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:19.286804 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:19.286732    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:19.288365 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:19.288334    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 01:54:21.961251 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:21.960150    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:22.135349 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:22.135305    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:22.216429 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:22.216330    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" podUID="6e39fc3d-d730-4df7-8d10-2376656a88d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:22.404381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:22.404334    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb" podUID="c9628178-6287-4ba3-9467-93f84ae883ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:22.968411 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:22.968357    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:22.969747 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:22.969723    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:54:23.378305 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:23.274547    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" podUID="6e39fc3d-d730-4df7-8d10-2376656a88d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:23.415249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:23.407820    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:54:23.451246 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:23.442393    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb" podUID="c9628178-6287-4ba3-9467-93f84ae883ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:23.609267 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:23.607349    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:54:23.655328 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:23.642738    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:54:23.784288 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:23.738146    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:54:24.063389 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:24.062284    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:54:24.299248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:24.284302    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:24.330019 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:24.329938    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97ffaad1-7m4qt"]
Nov 02 01:54:24.392600 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:24.384195    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97ffaad1-7m4qt"]
Nov 02 01:54:25.044842 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.044804    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d4c395d-b06a-460f-aabb-d5395e5293cf" path="/var/lib/kubelet/pods/0d4c395d-b06a-460f-aabb-d5395e5293cf/volumes"
Nov 02 01:54:25.046030 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.046006    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39063960-xvqdf"]
Nov 02 01:54:25.049060 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.049038    5875 cpu_manager.go:414] "containerMap: removing container" podUID="e4d60ff0-bf59-4022-a94f-1c3e560a7a6c" containerName="revoke-keys"
Nov 02 01:54:25.049203 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.049187    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4d60ff0-bf59-4022-a94f-1c3e560a7a6c" containerName="revoke-keys"
Nov 02 01:54:25.053867 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.053848    5875 cpu_manager.go:414] "containerMap: removing container" podUID="b9fe9e10-c020-4f72-a108-72cabf18cbae" containerName="revoke-keys"
Nov 02 01:54:25.053996 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.053983    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9fe9e10-c020-4f72-a108-72cabf18cbae" containerName="revoke-keys"
Nov 02 01:54:25.054409 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.054383    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9fe9e10-c020-4f72-a108-72cabf18cbae" containerName="revoke-keys"
Nov 02 01:54:25.054565 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.054553    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4d60ff0-bf59-4022-a94f-1c3e560a7a6c" containerName="revoke-keys"
Nov 02 01:54:25.066945 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.066919    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39063960-xvqdf"]
Nov 02 01:54:25.067375 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.066979    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3dadcad-dqq47"]
Nov 02 01:54:25.068586 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.068566    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39063960-xvqdf"
Nov 02 01:54:25.072853 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.072826    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3dadcad-dqq47"]
Nov 02 01:54:25.073118 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.073034    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dadcad-dqq47"
Nov 02 01:54:25.093810 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.093783    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd3773f-67de-479a-97ff-666544bcb640-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3dadcad-dqq47\" (UID: \"ddd3773f-67de-479a-97ff-666544bcb640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dadcad-dqq47"
Nov 02 01:54:25.093934 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.093918    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmnkm\" (UniqueName: \"kubernetes.io/projected/ddd3773f-67de-479a-97ff-666544bcb640-kube-api-access-wmnkm\") pod \"maas-api-revoke-sub-d3dadcad-dqq47\" (UID: \"ddd3773f-67de-479a-97ff-666544bcb640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dadcad-dqq47"
Nov 02 01:54:25.094051 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.094032    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4858d3ea-468d-447d-9f30-3cf259397bbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39063960-xvqdf\" (UID: \"4858d3ea-468d-447d-9f30-3cf259397bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39063960-xvqdf"
Nov 02 01:54:25.094126 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.094087    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnwdz\" (UniqueName: \"kubernetes.io/projected/4858d3ea-468d-447d-9f30-3cf259397bbb-kube-api-access-hnwdz\") pod \"maas-api-revoke-sub-39063960-xvqdf\" (UID: \"4858d3ea-468d-447d-9f30-3cf259397bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39063960-xvqdf"
Nov 02 01:54:25.094252 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.094201    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd3773f-67de-479a-97ff-666544bcb640-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3dadcad-dqq47\" (UID: \"ddd3773f-67de-479a-97ff-666544bcb640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dadcad-dqq47"
Nov 02 01:54:25.094322 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.094303    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4858d3ea-468d-447d-9f30-3cf259397bbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39063960-xvqdf\" (UID: \"4858d3ea-468d-447d-9f30-3cf259397bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39063960-xvqdf"
Nov 02 01:54:25.122245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.121270    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:54:25.122245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.121282    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:54:25.195363 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.195326    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd3773f-67de-479a-97ff-666544bcb640-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3dadcad-dqq47\" (UID: \"ddd3773f-67de-479a-97ff-666544bcb640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dadcad-dqq47"
Nov 02 01:54:25.195534 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.195431    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmnkm\" (UniqueName: \"kubernetes.io/projected/ddd3773f-67de-479a-97ff-666544bcb640-kube-api-access-wmnkm\") pod \"maas-api-revoke-sub-d3dadcad-dqq47\" (UID: \"ddd3773f-67de-479a-97ff-666544bcb640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dadcad-dqq47"
Nov 02 01:54:25.195534 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.195514    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4858d3ea-468d-447d-9f30-3cf259397bbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39063960-xvqdf\" (UID: \"4858d3ea-468d-447d-9f30-3cf259397bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39063960-xvqdf"
Nov 02 01:54:25.195677 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.195560    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnwdz\" (UniqueName: \"kubernetes.io/projected/4858d3ea-468d-447d-9f30-3cf259397bbb-kube-api-access-hnwdz\") pod \"maas-api-revoke-sub-39063960-xvqdf\" (UID: \"4858d3ea-468d-447d-9f30-3cf259397bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39063960-xvqdf"
Nov 02 01:54:25.195677 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.195660    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd3773f-67de-479a-97ff-666544bcb640-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3dadcad-dqq47\" (UID: \"ddd3773f-67de-479a-97ff-666544bcb640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dadcad-dqq47"
Nov 02 01:54:25.195782 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.195714    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4858d3ea-468d-447d-9f30-3cf259397bbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39063960-xvqdf\" (UID: \"4858d3ea-468d-447d-9f30-3cf259397bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39063960-xvqdf"
Nov 02 01:54:25.196377 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.196315    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4858d3ea-468d-447d-9f30-3cf259397bbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39063960-xvqdf\" (UID: \"4858d3ea-468d-447d-9f30-3cf259397bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39063960-xvqdf"
Nov 02 01:54:25.196477 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.196376    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd3773f-67de-479a-97ff-666544bcb640-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3dadcad-dqq47\" (UID: \"ddd3773f-67de-479a-97ff-666544bcb640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dadcad-dqq47"
Nov 02 01:54:25.198426 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.197922    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd3773f-67de-479a-97ff-666544bcb640-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3dadcad-dqq47\" (UID: \"ddd3773f-67de-479a-97ff-666544bcb640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dadcad-dqq47"
Nov 02 01:54:25.201691 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.201670    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4858d3ea-468d-447d-9f30-3cf259397bbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39063960-xvqdf\" (UID: \"4858d3ea-468d-447d-9f30-3cf259397bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39063960-xvqdf"
Nov 02 01:54:25.256435 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.256400    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnwdz\" (UniqueName: \"kubernetes.io/projected/4858d3ea-468d-447d-9f30-3cf259397bbb-kube-api-access-hnwdz\") pod \"maas-api-revoke-sub-39063960-xvqdf\" (UID: \"4858d3ea-468d-447d-9f30-3cf259397bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39063960-xvqdf"
Nov 02 01:54:25.256666 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.256644    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmnkm\" (UniqueName: \"kubernetes.io/projected/ddd3773f-67de-479a-97ff-666544bcb640-kube-api-access-wmnkm\") pod \"maas-api-revoke-sub-d3dadcad-dqq47\" (UID: \"ddd3773f-67de-479a-97ff-666544bcb640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dadcad-dqq47"
Nov 02 01:54:25.397806 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.397722    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39063960-xvqdf"
Nov 02 01:54:25.409053 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.409013    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dadcad-dqq47"
Nov 02 01:54:25.783655 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.783626    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39063960-xvqdf"]
Nov 02 01:54:25.783962 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:54:25.783930    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4858d3ea_468d_447d_9f30_3cf259397bbb.slice/crio-611186ac5b2c7a5772b12d4facb518aa649ac92f5e7626b7b873b22eeddc4cb8 WatchSource:0}: Error finding container 611186ac5b2c7a5772b12d4facb518aa649ac92f5e7626b7b873b22eeddc4cb8: Status 404 returned error can't find the container with id 611186ac5b2c7a5772b12d4facb518aa649ac92f5e7626b7b873b22eeddc4cb8
Nov 02 01:54:25.862425 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:25.861872    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3dadcad-dqq47"]
Nov 02 01:54:25.864315 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:54:25.864245    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd3773f_67de_479a_97ff_666544bcb640.slice/crio-3370449aa83fd2bb8668ef768d4dda44353734c847c236517c217d32acdb3635 WatchSource:0}: Error finding container 3370449aa83fd2bb8668ef768d4dda44353734c847c236517c217d32acdb3635: Status 404 returned error can't find the container with id 3370449aa83fd2bb8668ef768d4dda44353734c847c236517c217d32acdb3635
Nov 02 01:54:26.023435 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:26.023402    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dadcad-dqq47" event={"ID":"ddd3773f-67de-479a-97ff-666544bcb640","Type":"ContainerStarted","Data":"3370449aa83fd2bb8668ef768d4dda44353734c847c236517c217d32acdb3635"}
Nov 02 01:54:26.024782 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:26.024759    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39063960-xvqdf" event={"ID":"4858d3ea-468d-447d-9f30-3cf259397bbb","Type":"ContainerStarted","Data":"611186ac5b2c7a5772b12d4facb518aa649ac92f5e7626b7b873b22eeddc4cb8"}
Nov 02 01:54:27.034420 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:27.034384    5875 generic.go:356] "Generic (PLEG): container finished" podID="4858d3ea-468d-447d-9f30-3cf259397bbb" containerID="d626d488c15651e4a16ad010ea77207c0b3893bba47132eb478bc9b7ab9c232e" exitCode=0
Nov 02 01:54:27.034833 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:27.034461    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39063960-xvqdf" event={"ID":"4858d3ea-468d-447d-9f30-3cf259397bbb","Type":"ContainerDied","Data":"d626d488c15651e4a16ad010ea77207c0b3893bba47132eb478bc9b7ab9c232e"}
Nov 02 01:54:27.036173 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:27.036143    5875 generic.go:356] "Generic (PLEG): container finished" podID="ddd3773f-67de-479a-97ff-666544bcb640" containerID="d83100652a38b0cb03db6caef12b80701adf82cdff8abc5461326841a124ebe5" exitCode=0
Nov 02 01:54:27.036299 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:27.036188    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dadcad-dqq47" event={"ID":"ddd3773f-67de-479a-97ff-666544bcb640","Type":"ContainerDied","Data":"d83100652a38b0cb03db6caef12b80701adf82cdff8abc5461326841a124ebe5"}
Nov 02 01:54:28.214142 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:28.214107    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dadcad-dqq47"
Nov 02 01:54:28.248796 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:28.248773    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddd3773f-67de-479a-97ff-666544bcb640-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd3773f-67de-479a-97ff-666544bcb640-maas-api-openshift-ca\") pod \"ddd3773f-67de-479a-97ff-666544bcb640\" (UID: \"ddd3773f-67de-479a-97ff-666544bcb640\") "
Nov 02 01:54:28.248969 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:28.248955    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddd3773f-67de-479a-97ff-666544bcb640-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd3773f-67de-479a-97ff-666544bcb640-maas-api-xks-ca\") pod \"ddd3773f-67de-479a-97ff-666544bcb640\" (UID: \"ddd3773f-67de-479a-97ff-666544bcb640\") "
Nov 02 01:54:28.249013 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:28.248991    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddd3773f-67de-479a-97ff-666544bcb640-kube-api-access-wmnkm\" (UniqueName: \"kubernetes.io/projected/ddd3773f-67de-479a-97ff-666544bcb640-kube-api-access-wmnkm\") pod \"ddd3773f-67de-479a-97ff-666544bcb640\" (UID: \"ddd3773f-67de-479a-97ff-666544bcb640\") "
Nov 02 01:54:28.249202 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:28.249171    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd3773f-67de-479a-97ff-666544bcb640-maas-api-openshift-ca" pod "ddd3773f-67de-479a-97ff-666544bcb640" (UID: "ddd3773f-67de-479a-97ff-666544bcb640"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:28.249527 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:28.249506    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd3773f-67de-479a-97ff-666544bcb640-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:28.251066 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:28.251042    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddd3773f-67de-479a-97ff-666544bcb640-maas-api-xks-ca" pod "ddd3773f-67de-479a-97ff-666544bcb640" (UID: "ddd3773f-67de-479a-97ff-666544bcb640"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:28.251139 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:28.251100    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd3773f-67de-479a-97ff-666544bcb640-kube-api-access-wmnkm" pod "ddd3773f-67de-479a-97ff-666544bcb640" (UID: "ddd3773f-67de-479a-97ff-666544bcb640"). InnerVolumeSpecName "kube-api-access-wmnkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:28.267388 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:28.267365    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39063960-xvqdf"
Nov 02 01:54:28.350000 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:28.349909    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4858d3ea-468d-447d-9f30-3cf259397bbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4858d3ea-468d-447d-9f30-3cf259397bbb-maas-api-openshift-ca\") pod \"4858d3ea-468d-447d-9f30-3cf259397bbb\" (UID: \"4858d3ea-468d-447d-9f30-3cf259397bbb\") "
Nov 02 01:54:28.350165 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:28.350127    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4858d3ea-468d-447d-9f30-3cf259397bbb-kube-api-access-hnwdz\" (UniqueName: \"kubernetes.io/projected/4858d3ea-468d-447d-9f30-3cf259397bbb-kube-api-access-hnwdz\") pod \"4858d3ea-468d-447d-9f30-3cf259397bbb\" (UID: \"4858d3ea-468d-447d-9f30-3cf259397bbb\") "
Nov 02 01:54:28.350295 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:28.350279    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4858d3ea-468d-447d-9f30-3cf259397bbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4858d3ea-468d-447d-9f30-3cf259397bbb-maas-api-xks-ca\") pod \"4858d3ea-468d-447d-9f30-3cf259397bbb\" (UID: \"4858d3ea-468d-447d-9f30-3cf259397bbb\") "
Nov 02 01:54:28.350354 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:28.350291    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4858d3ea-468d-447d-9f30-3cf259397bbb-maas-api-openshift-ca" pod "4858d3ea-468d-447d-9f30-3cf259397bbb" (UID: "4858d3ea-468d-447d-9f30-3cf259397bbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:28.351117 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:28.351090    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd3773f-67de-479a-97ff-666544bcb640-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:28.351269 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:28.351119    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmnkm\" (UniqueName: \"kubernetes.io/projected/ddd3773f-67de-479a-97ff-666544bcb640-kube-api-access-wmnkm\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:28.351269 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:28.351138    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4858d3ea-468d-447d-9f30-3cf259397bbb-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:28.352426 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:28.352398    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4858d3ea-468d-447d-9f30-3cf259397bbb-maas-api-xks-ca" pod "4858d3ea-468d-447d-9f30-3cf259397bbb" (UID: "4858d3ea-468d-447d-9f30-3cf259397bbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:28.352525 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:28.352452    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4858d3ea-468d-447d-9f30-3cf259397bbb-kube-api-access-hnwdz" pod "4858d3ea-468d-447d-9f30-3cf259397bbb" (UID: "4858d3ea-468d-447d-9f30-3cf259397bbb"). InnerVolumeSpecName "kube-api-access-hnwdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:28.453120 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:28.453084    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnwdz\" (UniqueName: \"kubernetes.io/projected/4858d3ea-468d-447d-9f30-3cf259397bbb-kube-api-access-hnwdz\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:28.453120 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:28.453124    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4858d3ea-468d-447d-9f30-3cf259397bbb-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:29.051629 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.051544    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39063960-xvqdf"
Nov 02 01:54:29.051629 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.051573    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39063960-xvqdf" event={"ID":"4858d3ea-468d-447d-9f30-3cf259397bbb","Type":"ContainerDied","Data":"611186ac5b2c7a5772b12d4facb518aa649ac92f5e7626b7b873b22eeddc4cb8"}
Nov 02 01:54:29.051629 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.051620    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="611186ac5b2c7a5772b12d4facb518aa649ac92f5e7626b7b873b22eeddc4cb8"
Nov 02 01:54:29.053023 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.053000    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dadcad-dqq47" event={"ID":"ddd3773f-67de-479a-97ff-666544bcb640","Type":"ContainerDied","Data":"3370449aa83fd2bb8668ef768d4dda44353734c847c236517c217d32acdb3635"}
Nov 02 01:54:29.053023 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.053020    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dadcad-dqq47"
Nov 02 01:54:29.053185 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.053039    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3370449aa83fd2bb8668ef768d4dda44353734c847c236517c217d32acdb3635"
Nov 02 01:54:29.736091 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.736056    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7146e727-wzmhp"]
Nov 02 01:54:29.736701 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.736686    5875 cpu_manager.go:414] "containerMap: removing container" podUID="4858d3ea-468d-447d-9f30-3cf259397bbb" containerName="revoke-keys"
Nov 02 01:54:29.736701 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.736701    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4858d3ea-468d-447d-9f30-3cf259397bbb" containerName="revoke-keys"
Nov 02 01:54:29.736784 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.736735    5875 cpu_manager.go:414] "containerMap: removing container" podUID="ddd3773f-67de-479a-97ff-666544bcb640" containerName="revoke-keys"
Nov 02 01:54:29.736784 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.736741    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd3773f-67de-479a-97ff-666544bcb640" containerName="revoke-keys"
Nov 02 01:54:29.736855 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.736844    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd3773f-67de-479a-97ff-666544bcb640" containerName="revoke-keys"
Nov 02 01:54:29.736887 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.736857    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="4858d3ea-468d-447d-9f30-3cf259397bbb" containerName="revoke-keys"
Nov 02 01:54:29.740702 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.740682    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7146e727-wzmhp"
Nov 02 01:54:29.776658 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.776632    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a20c98-086f-4f64-9a5a-5e2a2d0714df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7146e727-wzmhp\" (UID: \"46a20c98-086f-4f64-9a5a-5e2a2d0714df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7146e727-wzmhp"
Nov 02 01:54:29.776880 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.776858    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a20c98-086f-4f64-9a5a-5e2a2d0714df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7146e727-wzmhp\" (UID: \"46a20c98-086f-4f64-9a5a-5e2a2d0714df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7146e727-wzmhp"
Nov 02 01:54:29.776984 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.776952    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nkh6\" (UniqueName: \"kubernetes.io/projected/46a20c98-086f-4f64-9a5a-5e2a2d0714df-kube-api-access-6nkh6\") pod \"maas-api-revoke-sub-7146e727-wzmhp\" (UID: \"46a20c98-086f-4f64-9a5a-5e2a2d0714df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7146e727-wzmhp"
Nov 02 01:54:29.793139 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.793109    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:54:29.798317 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.798277    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7146e727-wzmhp"]
Nov 02 01:54:29.842628 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.842596    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:54:29.878369 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.878329    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a20c98-086f-4f64-9a5a-5e2a2d0714df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7146e727-wzmhp\" (UID: \"46a20c98-086f-4f64-9a5a-5e2a2d0714df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7146e727-wzmhp"
Nov 02 01:54:29.878534 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.878396    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nkh6\" (UniqueName: \"kubernetes.io/projected/46a20c98-086f-4f64-9a5a-5e2a2d0714df-kube-api-access-6nkh6\") pod \"maas-api-revoke-sub-7146e727-wzmhp\" (UID: \"46a20c98-086f-4f64-9a5a-5e2a2d0714df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7146e727-wzmhp"
Nov 02 01:54:29.878607 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.878590    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a20c98-086f-4f64-9a5a-5e2a2d0714df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7146e727-wzmhp\" (UID: \"46a20c98-086f-4f64-9a5a-5e2a2d0714df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7146e727-wzmhp"
Nov 02 01:54:29.878990 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.878963    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a20c98-086f-4f64-9a5a-5e2a2d0714df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7146e727-wzmhp\" (UID: \"46a20c98-086f-4f64-9a5a-5e2a2d0714df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7146e727-wzmhp"
Nov 02 01:54:29.881210 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.881187    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a20c98-086f-4f64-9a5a-5e2a2d0714df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7146e727-wzmhp\" (UID: \"46a20c98-086f-4f64-9a5a-5e2a2d0714df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7146e727-wzmhp"
Nov 02 01:54:29.939171 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:29.939135    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nkh6\" (UniqueName: \"kubernetes.io/projected/46a20c98-086f-4f64-9a5a-5e2a2d0714df-kube-api-access-6nkh6\") pod \"maas-api-revoke-sub-7146e727-wzmhp\" (UID: \"46a20c98-086f-4f64-9a5a-5e2a2d0714df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7146e727-wzmhp"
Nov 02 01:54:30.055881 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:30.055576    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7146e727-wzmhp"
Nov 02 01:54:30.441855 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:30.441828    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7146e727-wzmhp"]
Nov 02 01:54:30.442678 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:54:30.442655    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a20c98_086f_4f64_9a5a_5e2a2d0714df.slice/crio-9ab391b637938995479f4e13bb15b65d9ad1d65de251d4dec023bcfea601963d WatchSource:0}: Error finding container 9ab391b637938995479f4e13bb15b65d9ad1d65de251d4dec023bcfea601963d: Status 404 returned error can't find the container with id 9ab391b637938995479f4e13bb15b65d9ad1d65de251d4dec023bcfea601963d
Nov 02 01:54:31.075619 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:31.075535    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7146e727-wzmhp" event={"ID":"46a20c98-086f-4f64-9a5a-5e2a2d0714df","Type":"ContainerStarted","Data":"d6d067f6035cd969943eee705018dd56d746f90fd424b2e5023bda752a4f2323"}
Nov 02 01:54:31.076014 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:31.075631    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7146e727-wzmhp" event={"ID":"46a20c98-086f-4f64-9a5a-5e2a2d0714df","Type":"ContainerStarted","Data":"9ab391b637938995479f4e13bb15b65d9ad1d65de251d4dec023bcfea601963d"}
Nov 02 01:54:31.138455 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:31.138427    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07ab7724-l4cr4"]
Nov 02 01:54:31.143071 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:31.143052    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ab7724-l4cr4"
Nov 02 01:54:31.193690 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:31.193643    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7146e727-wzmhp" podStartSLOduration=2.193629622 podStartE2EDuration="2.193629622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:31.191696552 +0000 UTC m=+1318.632733081" watchObservedRunningTime="2026-11-02 01:54:31.193629622 +0000 UTC m=+1318.634666119"
Nov 02 01:54:31.197898 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:31.197866    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d476\" (UniqueName: \"kubernetes.io/projected/0336b82f-b163-434d-952b-e37d9f8216a7-kube-api-access-4d476\") pod \"maas-api-revoke-sub-07ab7724-l4cr4\" (UID: \"0336b82f-b163-434d-952b-e37d9f8216a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ab7724-l4cr4"
Nov 02 01:54:31.198043 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:31.197935    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0336b82f-b163-434d-952b-e37d9f8216a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07ab7724-l4cr4\" (UID: \"0336b82f-b163-434d-952b-e37d9f8216a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ab7724-l4cr4"
Nov 02 01:54:31.198043 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:31.197963    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0336b82f-b163-434d-952b-e37d9f8216a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07ab7724-l4cr4\" (UID: \"0336b82f-b163-434d-952b-e37d9f8216a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ab7724-l4cr4"
Nov 02 01:54:31.201508 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:31.201480    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07ab7724-l4cr4"]
Nov 02 01:54:31.298950 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:31.298925    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d476\" (UniqueName: \"kubernetes.io/projected/0336b82f-b163-434d-952b-e37d9f8216a7-kube-api-access-4d476\") pod \"maas-api-revoke-sub-07ab7724-l4cr4\" (UID: \"0336b82f-b163-434d-952b-e37d9f8216a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ab7724-l4cr4"
Nov 02 01:54:31.299055 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:31.298996    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0336b82f-b163-434d-952b-e37d9f8216a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07ab7724-l4cr4\" (UID: \"0336b82f-b163-434d-952b-e37d9f8216a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ab7724-l4cr4"
Nov 02 01:54:31.299055 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:31.299021    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0336b82f-b163-434d-952b-e37d9f8216a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07ab7724-l4cr4\" (UID: \"0336b82f-b163-434d-952b-e37d9f8216a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ab7724-l4cr4"
Nov 02 01:54:31.299729 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:31.299706    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0336b82f-b163-434d-952b-e37d9f8216a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07ab7724-l4cr4\" (UID: \"0336b82f-b163-434d-952b-e37d9f8216a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ab7724-l4cr4"
Nov 02 01:54:31.301573 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:31.301553    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0336b82f-b163-434d-952b-e37d9f8216a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07ab7724-l4cr4\" (UID: \"0336b82f-b163-434d-952b-e37d9f8216a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ab7724-l4cr4"
Nov 02 01:54:31.359813 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:31.359765    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d476\" (UniqueName: \"kubernetes.io/projected/0336b82f-b163-434d-952b-e37d9f8216a7-kube-api-access-4d476\") pod \"maas-api-revoke-sub-07ab7724-l4cr4\" (UID: \"0336b82f-b163-434d-952b-e37d9f8216a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ab7724-l4cr4"
Nov 02 01:54:31.457939 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:31.457906    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ab7724-l4cr4"
Nov 02 01:54:31.837340 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:31.837311    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07ab7724-l4cr4"]
Nov 02 01:54:31.838372 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:54:31.838344    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0336b82f_b163_434d_952b_e37d9f8216a7.slice/crio-bf5f3c3e831caec630b22d5b273b995e4c51b9fb5d96a51a82498f406195e187 WatchSource:0}: Error finding container bf5f3c3e831caec630b22d5b273b995e4c51b9fb5d96a51a82498f406195e187: Status 404 returned error can't find the container with id bf5f3c3e831caec630b22d5b273b995e4c51b9fb5d96a51a82498f406195e187
Nov 02 01:54:32.084984 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:32.084890    5875 generic.go:356] "Generic (PLEG): container finished" podID="46a20c98-086f-4f64-9a5a-5e2a2d0714df" containerID="d6d067f6035cd969943eee705018dd56d746f90fd424b2e5023bda752a4f2323" exitCode=0
Nov 02 01:54:32.085390 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:32.084984    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7146e727-wzmhp" event={"ID":"46a20c98-086f-4f64-9a5a-5e2a2d0714df","Type":"ContainerDied","Data":"d6d067f6035cd969943eee705018dd56d746f90fd424b2e5023bda752a4f2323"}
Nov 02 01:54:32.086399 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:32.086376    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ab7724-l4cr4" event={"ID":"0336b82f-b163-434d-952b-e37d9f8216a7","Type":"ContainerStarted","Data":"bf5f3c3e831caec630b22d5b273b995e4c51b9fb5d96a51a82498f406195e187"}
Nov 02 01:54:32.889476 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:32.889447    5875 scope.go:122] "RemoveContainer" containerID="ddb6604510e3dce680ff827c6f49ad1cde01147e272fe780f8541dc7523df05f"
Nov 02 01:54:32.899628 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:32.899608    5875 scope.go:122] "RemoveContainer" containerID="37ee72a67ec7d6cf5e012e41acbfa3a01f609088720740dd110656ce964517c2"
Nov 02 01:54:32.911701 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:32.911679    5875 scope.go:122] "RemoveContainer" containerID="695b221e97ea7d7e9d61b412aa4d39052514b7afe95b1cd9999324276d7f7206"
Nov 02 01:54:32.921005 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:32.920988    5875 scope.go:122] "RemoveContainer" containerID="97dcab673d795af310f792476a95272af9c27433e7076e5d5f4b96a2dcae2a16"
Nov 02 01:54:32.929895 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:32.929875    5875 scope.go:122] "RemoveContainer" containerID="6793b50bbf0cdbf7e80b3e3f1f6106b81f93a4d06a6284c50e48ec8b2df9d1fa"
Nov 02 01:54:33.104814 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:33.104775    5875 generic.go:356] "Generic (PLEG): container finished" podID="0336b82f-b163-434d-952b-e37d9f8216a7" containerID="aa4da8b0d1745a98b37d1d9abedfcf31f22cc6455e00c7c914f61cce6884d65d" exitCode=0
Nov 02 01:54:33.105212 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:33.104859    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ab7724-l4cr4" event={"ID":"0336b82f-b163-434d-952b-e37d9f8216a7","Type":"ContainerDied","Data":"aa4da8b0d1745a98b37d1d9abedfcf31f22cc6455e00c7c914f61cce6884d65d"}
Nov 02 01:54:33.278564 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:33.278540    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7146e727-wzmhp"
Nov 02 01:54:33.427029 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:33.426997    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46a20c98-086f-4f64-9a5a-5e2a2d0714df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a20c98-086f-4f64-9a5a-5e2a2d0714df-maas-api-xks-ca\") pod \"46a20c98-086f-4f64-9a5a-5e2a2d0714df\" (UID: \"46a20c98-086f-4f64-9a5a-5e2a2d0714df\") "
Nov 02 01:54:33.427313 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:33.427291    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46a20c98-086f-4f64-9a5a-5e2a2d0714df-kube-api-access-6nkh6\" (UniqueName: \"kubernetes.io/projected/46a20c98-086f-4f64-9a5a-5e2a2d0714df-kube-api-access-6nkh6\") pod \"46a20c98-086f-4f64-9a5a-5e2a2d0714df\" (UID: \"46a20c98-086f-4f64-9a5a-5e2a2d0714df\") "
Nov 02 01:54:33.427426 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:33.427386    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46a20c98-086f-4f64-9a5a-5e2a2d0714df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a20c98-086f-4f64-9a5a-5e2a2d0714df-maas-api-openshift-ca\") pod \"46a20c98-086f-4f64-9a5a-5e2a2d0714df\" (UID: \"46a20c98-086f-4f64-9a5a-5e2a2d0714df\") "
Nov 02 01:54:33.427917 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:33.427888    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a20c98-086f-4f64-9a5a-5e2a2d0714df-maas-api-openshift-ca" pod "46a20c98-086f-4f64-9a5a-5e2a2d0714df" (UID: "46a20c98-086f-4f64-9a5a-5e2a2d0714df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:33.429807 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:33.429781    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46a20c98-086f-4f64-9a5a-5e2a2d0714df-maas-api-xks-ca" pod "46a20c98-086f-4f64-9a5a-5e2a2d0714df" (UID: "46a20c98-086f-4f64-9a5a-5e2a2d0714df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:33.430059 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:33.430038    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a20c98-086f-4f64-9a5a-5e2a2d0714df-kube-api-access-6nkh6" pod "46a20c98-086f-4f64-9a5a-5e2a2d0714df" (UID: "46a20c98-086f-4f64-9a5a-5e2a2d0714df"). InnerVolumeSpecName "kube-api-access-6nkh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:33.529396 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:33.529316    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nkh6\" (UniqueName: \"kubernetes.io/projected/46a20c98-086f-4f64-9a5a-5e2a2d0714df-kube-api-access-6nkh6\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.529396 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:33.529343    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a20c98-086f-4f64-9a5a-5e2a2d0714df-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:33.529396 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:33.529354    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a20c98-086f-4f64-9a5a-5e2a2d0714df-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.112635 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.112567    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7146e727-wzmhp"
Nov 02 01:54:34.112635 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.112599    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7146e727-wzmhp" event={"ID":"46a20c98-086f-4f64-9a5a-5e2a2d0714df","Type":"ContainerDied","Data":"9ab391b637938995479f4e13bb15b65d9ad1d65de251d4dec023bcfea601963d"}
Nov 02 01:54:34.113089 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.112642    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ab391b637938995479f4e13bb15b65d9ad1d65de251d4dec023bcfea601963d"
Nov 02 01:54:34.278649 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.278625    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ab7724-l4cr4"
Nov 02 01:54:34.340364 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.340337    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0336b82f-b163-434d-952b-e37d9f8216a7-kube-api-access-4d476\" (UniqueName: \"kubernetes.io/projected/0336b82f-b163-434d-952b-e37d9f8216a7-kube-api-access-4d476\") pod \"0336b82f-b163-434d-952b-e37d9f8216a7\" (UID: \"0336b82f-b163-434d-952b-e37d9f8216a7\") "
Nov 02 01:54:34.340496 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.340455    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0336b82f-b163-434d-952b-e37d9f8216a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0336b82f-b163-434d-952b-e37d9f8216a7-maas-api-openshift-ca\") pod \"0336b82f-b163-434d-952b-e37d9f8216a7\" (UID: \"0336b82f-b163-434d-952b-e37d9f8216a7\") "
Nov 02 01:54:34.340543 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.340508    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0336b82f-b163-434d-952b-e37d9f8216a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0336b82f-b163-434d-952b-e37d9f8216a7-maas-api-xks-ca\") pod \"0336b82f-b163-434d-952b-e37d9f8216a7\" (UID: \"0336b82f-b163-434d-952b-e37d9f8216a7\") "
Nov 02 01:54:34.340851 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.340822    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0336b82f-b163-434d-952b-e37d9f8216a7-maas-api-openshift-ca" pod "0336b82f-b163-434d-952b-e37d9f8216a7" (UID: "0336b82f-b163-434d-952b-e37d9f8216a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:34.342493 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.342470    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0336b82f-b163-434d-952b-e37d9f8216a7-kube-api-access-4d476" pod "0336b82f-b163-434d-952b-e37d9f8216a7" (UID: "0336b82f-b163-434d-952b-e37d9f8216a7"). InnerVolumeSpecName "kube-api-access-4d476". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:34.342606 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.342562    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0336b82f-b163-434d-952b-e37d9f8216a7-maas-api-xks-ca" pod "0336b82f-b163-434d-952b-e37d9f8216a7" (UID: "0336b82f-b163-434d-952b-e37d9f8216a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:34.442463 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.442382    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d476\" (UniqueName: \"kubernetes.io/projected/0336b82f-b163-434d-952b-e37d9f8216a7-kube-api-access-4d476\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.442463 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.442420    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0336b82f-b163-434d-952b-e37d9f8216a7-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.442463 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.442435    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0336b82f-b163-434d-952b-e37d9f8216a7-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:34.746160 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.746130    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"]
Nov 02 01:54:34.746752 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.746736    5875 cpu_manager.go:414] "containerMap: removing container" podUID="0336b82f-b163-434d-952b-e37d9f8216a7" containerName="revoke-keys"
Nov 02 01:54:34.746752 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.746750    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0336b82f-b163-434d-952b-e37d9f8216a7" containerName="revoke-keys"
Nov 02 01:54:34.746917 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.746787    5875 cpu_manager.go:414] "containerMap: removing container" podUID="46a20c98-086f-4f64-9a5a-5e2a2d0714df" containerName="revoke-keys"
Nov 02 01:54:34.746917 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.746792    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46a20c98-086f-4f64-9a5a-5e2a2d0714df" containerName="revoke-keys"
Nov 02 01:54:34.746917 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.746902    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="0336b82f-b163-434d-952b-e37d9f8216a7" containerName="revoke-keys"
Nov 02 01:54:34.747188 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.746924    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="46a20c98-086f-4f64-9a5a-5e2a2d0714df" containerName="revoke-keys"
Nov 02 01:54:34.750791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.750773    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:34.848846 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.848780    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpg5x\" (UniqueName: \"kubernetes.io/projected/3c4cedf1-b41a-42b5-9b5f-880b8264d434-kube-api-access-jpg5x\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:34.849028 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.848849    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-envoy\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:34.849028 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.848878    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-data\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:34.849028 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.848912    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-workload-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:34.849028 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.848940    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-token\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:34.849028 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.848979    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-workload-certs\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:34.849028 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.849019    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-credential-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:34.849456 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.849056    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-podinfo\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:34.849456 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.849091    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istiod-ca-cert\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:34.856915 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.856889    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"]
Nov 02 01:54:34.950604 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.950571    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpg5x\" (UniqueName: \"kubernetes.io/projected/3c4cedf1-b41a-42b5-9b5f-880b8264d434-kube-api-access-jpg5x\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:34.950792 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.950620    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-envoy\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:34.950792 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.950639    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-data\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:34.950792 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.950672    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-workload-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:34.950792 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.950694    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-token\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:34.950792 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.950725    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-workload-certs\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:34.950792 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.950766    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-credential-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:34.951106 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.950879    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-podinfo\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:34.951106 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.950931    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istiod-ca-cert\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:34.951213 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.951106    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-workload-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:34.951213 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.951120    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-workload-certs\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:34.951356 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.951276    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-data\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:34.951356 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.951308    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-credential-socket\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:34.951813 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.951761    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istiod-ca-cert\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:34.953008 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.952990    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-envoy\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:34.953436 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:34.953420    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-podinfo\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:35.010989 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:35.010925    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-token\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:35.011160 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:35.011142    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpg5x\" (UniqueName: \"kubernetes.io/projected/3c4cedf1-b41a-42b5-9b5f-880b8264d434-kube-api-access-jpg5x\") pod \"e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") " pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:35.063948 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:35.063924    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:35.126991 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:35.126936    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ab7724-l4cr4" event={"ID":"0336b82f-b163-434d-952b-e37d9f8216a7","Type":"ContainerDied","Data":"bf5f3c3e831caec630b22d5b273b995e4c51b9fb5d96a51a82498f406195e187"}
Nov 02 01:54:35.127385 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:35.127003    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf5f3c3e831caec630b22d5b273b995e4c51b9fb5d96a51a82498f406195e187"
Nov 02 01:54:35.127385 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:35.126955    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07ab7724-l4cr4"
Nov 02 01:54:35.524258 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:35.524213    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"]
Nov 02 01:54:35.525071 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:54:35.525049    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4cedf1_b41a_42b5_9b5f_880b8264d434.slice/crio-93406a1e36afed6c0e45b7e2d96335ca9a17d8a62511281f2f0ddb0dcd4f2ff2 WatchSource:0}: Error finding container 93406a1e36afed6c0e45b7e2d96335ca9a17d8a62511281f2f0ddb0dcd4f2ff2: Status 404 returned error can't find the container with id 93406a1e36afed6c0e45b7e2d96335ca9a17d8a62511281f2f0ddb0dcd4f2ff2
Nov 02 01:54:35.527713 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:35.527679    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:54:35.527818 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:35.527745    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:54:35.527818 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:35.527784    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:54:36.137542 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:36.137502    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt" event={"ID":"3c4cedf1-b41a-42b5-9b5f-880b8264d434","Type":"ContainerStarted","Data":"7125ca9a76e5525938ff5eb853b116962bc21b8375c3319c50ad929221d535d9"}
Nov 02 01:54:36.137974 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:36.137566    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt" event={"ID":"3c4cedf1-b41a-42b5-9b5f-880b8264d434","Type":"ContainerStarted","Data":"93406a1e36afed6c0e45b7e2d96335ca9a17d8a62511281f2f0ddb0dcd4f2ff2"}
Nov 02 01:54:36.358948 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:36.358895    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt" podStartSLOduration=2.358882996 podStartE2EDuration="2.358882996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:36.309058093 +0000 UTC m=+1323.750094592" watchObservedRunningTime="2026-11-02 01:54:36.358882996 +0000 UTC m=+1323.799919497"
Nov 02 01:54:37.064315 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:37.064273    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:37.066012 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:37.065983    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 01:54:37.066131 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:37.066040    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt" podUID="3c4cedf1-b41a-42b5-9b5f-880b8264d434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 01:54:38.065306 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.065271    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 01:54:38.065703 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.065327    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt" podUID="3c4cedf1-b41a-42b5-9b5f-880b8264d434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 01:54:38.525619 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.525578    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"]
Nov 02 01:54:38.530425 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.530400    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.629325 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.629290    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"]
Nov 02 01:54:38.640491 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.640455    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54jjw\" (UniqueName: \"kubernetes.io/projected/84804eea-7177-41c0-9bd1-63588f418caa-kube-api-access-54jjw\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.640700 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.640618    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-workload-certs\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.640793 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.640770    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84804eea-7177-41c0-9bd1-63588f418caa-istio-podinfo\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.640889 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.640868    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-istio-envoy\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.640948 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.640914    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84804eea-7177-41c0-9bd1-63588f418caa-istio-token\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.641010 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.640989    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-istio-data\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.641067 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.641023    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-credential-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.641119 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.641066    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84804eea-7177-41c0-9bd1-63588f418caa-istiod-ca-cert\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.641119 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.641096    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-workload-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.742812 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.742780    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54jjw\" (UniqueName: \"kubernetes.io/projected/84804eea-7177-41c0-9bd1-63588f418caa-kube-api-access-54jjw\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.743016 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.742854    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-workload-certs\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.743016 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.742912    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84804eea-7177-41c0-9bd1-63588f418caa-istio-podinfo\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.743016 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.742960    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-istio-envoy\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.743016 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.742991    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84804eea-7177-41c0-9bd1-63588f418caa-istio-token\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.743273 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.743021    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-istio-data\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.743273 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.743052    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-credential-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.743273 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.743092    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84804eea-7177-41c0-9bd1-63588f418caa-istiod-ca-cert\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.743273 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.743123    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-workload-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.743514 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.743486    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-istio-data\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.743582 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.743512    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-credential-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.743686 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.743661    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-workload-socket\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.743754 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.743693    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-workload-certs\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.744100 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.744071    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84804eea-7177-41c0-9bd1-63588f418caa-istiod-ca-cert\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.745767 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.745741    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84804eea-7177-41c0-9bd1-63588f418caa-istio-podinfo\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.745891 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.745871    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-istio-envoy\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.823141 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.823050    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84804eea-7177-41c0-9bd1-63588f418caa-istio-token\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.824535 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.824508    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54jjw\" (UniqueName: \"kubernetes.io/projected/84804eea-7177-41c0-9bd1-63588f418caa-kube-api-access-54jjw\") pod \"e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") " pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:38.844474 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:38.844435    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:39.064953 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:39.064912    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 01:54:39.065149 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:39.064971    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt" podUID="3c4cedf1-b41a-42b5-9b5f-880b8264d434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 01:54:39.324633 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:39.321675    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"]
Nov 02 01:54:39.328806 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:39.328771    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:54:39.328911 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:39.328837    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:54:39.328911 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:39.328865    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:54:40.066237 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:40.064934    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 01:54:40.066237 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:40.065002    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt" podUID="3c4cedf1-b41a-42b5-9b5f-880b8264d434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 01:54:40.168782 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:40.168744    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk" event={"ID":"84804eea-7177-41c0-9bd1-63588f418caa","Type":"ContainerStarted","Data":"8919c2fa00fc8b78ce9a93f2089fae2c5a27aff9b1d650ece888d2c602e46044"}
Nov 02 01:54:40.169039 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:40.169022    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk" event={"ID":"84804eea-7177-41c0-9bd1-63588f418caa","Type":"ContainerStarted","Data":"e47f45192d6bcb58187db915b99b968d4bfe7e12bc8f092a33d11ae8febd0887"}
Nov 02 01:54:40.852473 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:40.852395    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:40.854746 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:40.854709    5875 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 02 01:54:40.854884 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:40.854785    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk" podUID="84804eea-7177-41c0-9bd1-63588f418caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 02 01:54:40.931247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:40.930409    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk" podStartSLOduration=2.930386344 podStartE2EDuration="2.930386344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:40.352129666 +0000 UTC m=+1327.793166203" watchObservedRunningTime="2026-11-02 01:54:40.930386344 +0000 UTC m=+1328.371422870"
Nov 02 01:54:40.939279 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:40.931921    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-761c34e8-7k6dh"]
Nov 02 01:54:40.954263 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:40.954210    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-761c34e8-7k6dh"
Nov 02 01:54:40.962866 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:40.962530    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-761c34e8-7k6dh"]
Nov 02 01:54:41.021266 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:41.017580    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:54:41.021266 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:41.017711    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:54:41.060600 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:41.060389    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0610139-0aec-49ea-a67f-e7815c2f6fe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-761c34e8-7k6dh\" (UID: \"f0610139-0aec-49ea-a67f-e7815c2f6fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-761c34e8-7k6dh"
Nov 02 01:54:41.060600 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:41.060540    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f6q5\" (UniqueName: \"kubernetes.io/projected/f0610139-0aec-49ea-a67f-e7815c2f6fe4-kube-api-access-4f6q5\") pod \"maas-api-revoke-sub-761c34e8-7k6dh\" (UID: \"f0610139-0aec-49ea-a67f-e7815c2f6fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-761c34e8-7k6dh"
Nov 02 01:54:41.060859 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:41.060709    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0610139-0aec-49ea-a67f-e7815c2f6fe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-761c34e8-7k6dh\" (UID: \"f0610139-0aec-49ea-a67f-e7815c2f6fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-761c34e8-7k6dh"
Nov 02 01:54:41.065450 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:41.065090    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 01:54:41.065450 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:41.065138    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt" podUID="3c4cedf1-b41a-42b5-9b5f-880b8264d434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 01:54:41.161991 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:41.161900    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f6q5\" (UniqueName: \"kubernetes.io/projected/f0610139-0aec-49ea-a67f-e7815c2f6fe4-kube-api-access-4f6q5\") pod \"maas-api-revoke-sub-761c34e8-7k6dh\" (UID: \"f0610139-0aec-49ea-a67f-e7815c2f6fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-761c34e8-7k6dh"
Nov 02 01:54:41.162164 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:41.162130    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0610139-0aec-49ea-a67f-e7815c2f6fe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-761c34e8-7k6dh\" (UID: \"f0610139-0aec-49ea-a67f-e7815c2f6fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-761c34e8-7k6dh"
Nov 02 01:54:41.162523 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:41.162504    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0610139-0aec-49ea-a67f-e7815c2f6fe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-761c34e8-7k6dh\" (UID: \"f0610139-0aec-49ea-a67f-e7815c2f6fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-761c34e8-7k6dh"
Nov 02 01:54:41.163072 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:41.163053    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0610139-0aec-49ea-a67f-e7815c2f6fe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-761c34e8-7k6dh\" (UID: \"f0610139-0aec-49ea-a67f-e7815c2f6fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-761c34e8-7k6dh"
Nov 02 01:54:41.165088 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:41.165068    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0610139-0aec-49ea-a67f-e7815c2f6fe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-761c34e8-7k6dh\" (UID: \"f0610139-0aec-49ea-a67f-e7815c2f6fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-761c34e8-7k6dh"
Nov 02 01:54:41.251403 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:41.251366    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f6q5\" (UniqueName: \"kubernetes.io/projected/f0610139-0aec-49ea-a67f-e7815c2f6fe4-kube-api-access-4f6q5\") pod \"maas-api-revoke-sub-761c34e8-7k6dh\" (UID: \"f0610139-0aec-49ea-a67f-e7815c2f6fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-761c34e8-7k6dh"
Nov 02 01:54:41.272251 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:41.272199    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-761c34e8-7k6dh"
Nov 02 01:54:41.382311 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:54:41.382268    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:54:41.710482 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:41.710445    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-761c34e8-7k6dh"]
Nov 02 01:54:41.845507 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:41.845475    5875 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 02 01:54:41.845664 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:41.845525    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk" podUID="84804eea-7177-41c0-9bd1-63588f418caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 02 01:54:42.065494 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:42.065404    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 02 01:54:42.065494 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:42.065463    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt" podUID="3c4cedf1-b41a-42b5-9b5f-880b8264d434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 02 01:54:42.187952 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:42.187909    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-761c34e8-7k6dh" event={"ID":"f0610139-0aec-49ea-a67f-e7815c2f6fe4","Type":"ContainerStarted","Data":"a3f6e62a9287c235db4a0d12967801625c3deebd7af0a5ec1be58b7ed8a2b98f"}
Nov 02 01:54:42.844827 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:42.844799    5875 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 02 01:54:42.844977 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:42.844850    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk" podUID="84804eea-7177-41c0-9bd1-63588f418caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 02 01:54:43.132626 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:43.132536    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:43.133074 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:43.132959    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:43.133800 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:43.133779    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 01:54:43.196130 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:43.196098    5875 generic.go:356] "Generic (PLEG): container finished" podID="f0610139-0aec-49ea-a67f-e7815c2f6fe4" containerID="504e451ff8c696d115cbccefe6cb8f7dd99fab94228540954cfb90f806b5ae74" exitCode=0
Nov 02 01:54:43.196322 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:43.196183    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-761c34e8-7k6dh" event={"ID":"f0610139-0aec-49ea-a67f-e7815c2f6fe4","Type":"ContainerDied","Data":"504e451ff8c696d115cbccefe6cb8f7dd99fab94228540954cfb90f806b5ae74"}
Nov 02 01:54:43.845059 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:43.845028    5875 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 02 01:54:43.845266 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:43.845078    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk" podUID="84804eea-7177-41c0-9bd1-63588f418caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 02 01:54:44.436343 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:44.436317    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-761c34e8-7k6dh"
Nov 02 01:54:44.533319 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:44.533233    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0610139-0aec-49ea-a67f-e7815c2f6fe4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0610139-0aec-49ea-a67f-e7815c2f6fe4-maas-api-openshift-ca\") pod \"f0610139-0aec-49ea-a67f-e7815c2f6fe4\" (UID: \"f0610139-0aec-49ea-a67f-e7815c2f6fe4\") "
Nov 02 01:54:44.533493 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:44.533359    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0610139-0aec-49ea-a67f-e7815c2f6fe4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0610139-0aec-49ea-a67f-e7815c2f6fe4-maas-api-xks-ca\") pod \"f0610139-0aec-49ea-a67f-e7815c2f6fe4\" (UID: \"f0610139-0aec-49ea-a67f-e7815c2f6fe4\") "
Nov 02 01:54:44.533493 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:44.533431    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0610139-0aec-49ea-a67f-e7815c2f6fe4-kube-api-access-4f6q5\" (UniqueName: \"kubernetes.io/projected/f0610139-0aec-49ea-a67f-e7815c2f6fe4-kube-api-access-4f6q5\") pod \"f0610139-0aec-49ea-a67f-e7815c2f6fe4\" (UID: \"f0610139-0aec-49ea-a67f-e7815c2f6fe4\") "
Nov 02 01:54:44.533653 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:44.533625    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0610139-0aec-49ea-a67f-e7815c2f6fe4-maas-api-openshift-ca" pod "f0610139-0aec-49ea-a67f-e7815c2f6fe4" (UID: "f0610139-0aec-49ea-a67f-e7815c2f6fe4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:44.534304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:44.534236    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0610139-0aec-49ea-a67f-e7815c2f6fe4-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:44.535768 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:44.535742    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0610139-0aec-49ea-a67f-e7815c2f6fe4-kube-api-access-4f6q5" pod "f0610139-0aec-49ea-a67f-e7815c2f6fe4" (UID: "f0610139-0aec-49ea-a67f-e7815c2f6fe4"). InnerVolumeSpecName "kube-api-access-4f6q5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:44.536147 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:44.536127    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0610139-0aec-49ea-a67f-e7815c2f6fe4-maas-api-xks-ca" pod "f0610139-0aec-49ea-a67f-e7815c2f6fe4" (UID: "f0610139-0aec-49ea-a67f-e7815c2f6fe4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:44.635427 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:44.635393    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0610139-0aec-49ea-a67f-e7815c2f6fe4-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:44.635427 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:44.635424    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f6q5\" (UniqueName: \"kubernetes.io/projected/f0610139-0aec-49ea-a67f-e7815c2f6fe4-kube-api-access-4f6q5\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:44.762629 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:44.762595    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"]
Nov 02 01:54:44.762850 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:44.762827    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9" podUID="bafe4c70-f1ef-423f-a94c-552f5545ad6d" containerName="istio-proxy" containerID="cri-o://869871651f445395b7e5f1f979bee2a3b2a358a38df394218c5b1d028cd35e41" gracePeriod=30
Nov 02 01:54:44.766414 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:44.766390    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"]
Nov 02 01:54:44.766574 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:44.766557    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5" podUID="e1d7b68a-92f5-47c6-bedb-d75802e2232d" containerName="istio-proxy" containerID="cri-o://940bc71f358809f78adf665b284dcf6feefd4b3c3e6490e71c83c493a11008cf" gracePeriod=30
Nov 02 01:54:44.845105 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:44.845025    5875 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 02 01:54:44.845105 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:44.845090    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk" podUID="84804eea-7177-41c0-9bd1-63588f418caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 02 01:54:45.282249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:45.281978    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-761c34e8-7k6dh" event={"ID":"f0610139-0aec-49ea-a67f-e7815c2f6fe4","Type":"ContainerDied","Data":"a3f6e62a9287c235db4a0d12967801625c3deebd7af0a5ec1be58b7ed8a2b98f"}
Nov 02 01:54:45.282249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:45.282047    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3f6e62a9287c235db4a0d12967801625c3deebd7af0a5ec1be58b7ed8a2b98f"
Nov 02 01:54:45.282249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:45.282121    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-761c34e8-7k6dh"
Nov 02 01:54:45.845537 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:45.845504    5875 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 02 01:54:45.845931 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:45.845558    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk" podUID="84804eea-7177-41c0-9bd1-63588f418caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 02 01:54:46.848475 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:46.848441    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:46.848956 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:46.848941    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:46.849640 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:46.849620    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 01:54:48.806429 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:48.806397    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-gcng9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:54:48.806806 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:48.806474    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9" podUID="bafe4c70-f1ef-423f-a94c-552f5545ad6d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:54:50.154332 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.154305    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:54:50.206826 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.206804    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:54:50.300317 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.300289    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istiod-ca-cert\") pod \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") "
Nov 02 01:54:50.300497 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.300331    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-workload-certs\") pod \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") "
Nov 02 01:54:50.300497 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.300364    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-data\") pod \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") "
Nov 02 01:54:50.300613 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.300551    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-data\") pod \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") "
Nov 02 01:54:50.300613 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.300594    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-credential-socket\") pod \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") "
Nov 02 01:54:50.300707 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.300667    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-workload-certs" pod "e1d7b68a-92f5-47c6-bedb-d75802e2232d" (UID: "e1d7b68a-92f5-47c6-bedb-d75802e2232d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:50.300751 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.300707    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-data" pod "e1d7b68a-92f5-47c6-bedb-d75802e2232d" (UID: "e1d7b68a-92f5-47c6-bedb-d75802e2232d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:50.300804 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.300757    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istiod-ca-cert\") pod \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") "
Nov 02 01:54:50.300804 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.300790    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-podinfo\") pod \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") "
Nov 02 01:54:50.300913 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.300803    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istiod-ca-cert" pod "e1d7b68a-92f5-47c6-bedb-d75802e2232d" (UID: "e1d7b68a-92f5-47c6-bedb-d75802e2232d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:50.300913 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.300817    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-data" pod "bafe4c70-f1ef-423f-a94c-552f5545ad6d" (UID: "bafe4c70-f1ef-423f-a94c-552f5545ad6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:50.300913 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.300876    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-credential-socket" pod "e1d7b68a-92f5-47c6-bedb-d75802e2232d" (UID: "e1d7b68a-92f5-47c6-bedb-d75802e2232d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:50.300913 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.300872    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-workload-certs\") pod \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") "
Nov 02 01:54:50.301119 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.300921    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bafe4c70-f1ef-423f-a94c-552f5545ad6d-kube-api-access-b789m\" (UniqueName: \"kubernetes.io/projected/bafe4c70-f1ef-423f-a94c-552f5545ad6d-kube-api-access-b789m\") pod \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") "
Nov 02 01:54:50.301119 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.300961    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-envoy\") pod \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") "
Nov 02 01:54:50.301119 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.301016    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-workload-socket\") pod \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") "
Nov 02 01:54:50.301119 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.301051    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-workload-socket\") pod \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") "
Nov 02 01:54:50.301119 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.301080    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-workload-certs" pod "bafe4c70-f1ef-423f-a94c-552f5545ad6d" (UID: "bafe4c70-f1ef-423f-a94c-552f5545ad6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:50.301119 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.301093    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-envoy\") pod \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") "
Nov 02 01:54:50.301458 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.301123    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-podinfo\") pod \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") "
Nov 02 01:54:50.301458 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.301132    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istiod-ca-cert" pod "bafe4c70-f1ef-423f-a94c-552f5545ad6d" (UID: "bafe4c70-f1ef-423f-a94c-552f5545ad6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:50.301458 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.301169    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-credential-socket\") pod \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") "
Nov 02 01:54:50.301458 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.301265    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1d7b68a-92f5-47c6-bedb-d75802e2232d-kube-api-access-7pt4w\" (UniqueName: \"kubernetes.io/projected/e1d7b68a-92f5-47c6-bedb-d75802e2232d-kube-api-access-7pt4w\") pod \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") "
Nov 02 01:54:50.301458 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.301274    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-workload-socket" pod "bafe4c70-f1ef-423f-a94c-552f5545ad6d" (UID: "bafe4c70-f1ef-423f-a94c-552f5545ad6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:50.301458 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.301281    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-workload-socket" pod "e1d7b68a-92f5-47c6-bedb-d75802e2232d" (UID: "e1d7b68a-92f5-47c6-bedb-d75802e2232d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:50.301458 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.301291    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-token\" (UniqueName: \"kubernetes.io/projected/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-token\") pod \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\" (UID: \"e1d7b68a-92f5-47c6-bedb-d75802e2232d\") "
Nov 02 01:54:50.301458 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.301381    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-token\" (UniqueName: \"kubernetes.io/projected/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-token\") pod \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\" (UID: \"bafe4c70-f1ef-423f-a94c-552f5545ad6d\") "
Nov 02 01:54:50.302034 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.301479    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-credential-socket" pod "bafe4c70-f1ef-423f-a94c-552f5545ad6d" (UID: "bafe4c70-f1ef-423f-a94c-552f5545ad6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:50.302198 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.302146    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:50.302629 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.302164    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:50.302629 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.302507    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:50.302629 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.302526    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:50.302629 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.302540    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:50.302629 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.302553    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:50.302629 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.302567    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:50.302629 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.302582    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:50.302629 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.302595    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:50.302629 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.302609    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:50.303964 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.303930    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-podinfo" pod "bafe4c70-f1ef-423f-a94c-552f5545ad6d" (UID: "bafe4c70-f1ef-423f-a94c-552f5545ad6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:54:50.304175 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.304136    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-envoy" pod "bafe4c70-f1ef-423f-a94c-552f5545ad6d" (UID: "bafe4c70-f1ef-423f-a94c-552f5545ad6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:50.304280 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.304197    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-podinfo" pod "e1d7b68a-92f5-47c6-bedb-d75802e2232d" (UID: "e1d7b68a-92f5-47c6-bedb-d75802e2232d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:54:50.305283 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.305259    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-token" pod "bafe4c70-f1ef-423f-a94c-552f5545ad6d" (UID: "bafe4c70-f1ef-423f-a94c-552f5545ad6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:50.305374 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.305275    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-envoy" pod "e1d7b68a-92f5-47c6-bedb-d75802e2232d" (UID: "e1d7b68a-92f5-47c6-bedb-d75802e2232d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:50.305374 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.305295    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bafe4c70-f1ef-423f-a94c-552f5545ad6d-kube-api-access-b789m" pod "bafe4c70-f1ef-423f-a94c-552f5545ad6d" (UID: "bafe4c70-f1ef-423f-a94c-552f5545ad6d"). InnerVolumeSpecName "kube-api-access-b789m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:50.305374 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.305322    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1d7b68a-92f5-47c6-bedb-d75802e2232d-kube-api-access-7pt4w" pod "e1d7b68a-92f5-47c6-bedb-d75802e2232d" (UID: "e1d7b68a-92f5-47c6-bedb-d75802e2232d"). InnerVolumeSpecName "kube-api-access-7pt4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:50.305475 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.305443    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-token" pod "e1d7b68a-92f5-47c6-bedb-d75802e2232d" (UID: "e1d7b68a-92f5-47c6-bedb-d75802e2232d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:50.327555 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.327527    5875 generic.go:356] "Generic (PLEG): container finished" podID="bafe4c70-f1ef-423f-a94c-552f5545ad6d" containerID="869871651f445395b7e5f1f979bee2a3b2a358a38df394218c5b1d028cd35e41" exitCode=0
Nov 02 01:54:50.327772 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.327747    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9" event={"ID":"bafe4c70-f1ef-423f-a94c-552f5545ad6d","Type":"ContainerDied","Data":"869871651f445395b7e5f1f979bee2a3b2a358a38df394218c5b1d028cd35e41"}
Nov 02 01:54:50.327871 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.327809    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9" event={"ID":"bafe4c70-f1ef-423f-a94c-552f5545ad6d","Type":"ContainerDied","Data":"e48cf041dece4a656fb2f91104130fc588eca2b4eea95cd8792444edd03b8860"}
Nov 02 01:54:50.327871 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.327839    5875 scope.go:122] "RemoveContainer" containerID="869871651f445395b7e5f1f979bee2a3b2a358a38df394218c5b1d028cd35e41"
Nov 02 01:54:50.328009 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.327994    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"
Nov 02 01:54:50.333362 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.333332    5875 generic.go:356] "Generic (PLEG): container finished" podID="e1d7b68a-92f5-47c6-bedb-d75802e2232d" containerID="940bc71f358809f78adf665b284dcf6feefd4b3c3e6490e71c83c493a11008cf" exitCode=0
Nov 02 01:54:50.333524 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.333486    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5" event={"ID":"e1d7b68a-92f5-47c6-bedb-d75802e2232d","Type":"ContainerDied","Data":"940bc71f358809f78adf665b284dcf6feefd4b3c3e6490e71c83c493a11008cf"}
Nov 02 01:54:50.333616 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.333534    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5" event={"ID":"e1d7b68a-92f5-47c6-bedb-d75802e2232d","Type":"ContainerDied","Data":"cac892884e06abadbd425a42d0035ed9c27cbd89e6d3c36b7e0baf136f0209f1"}
Nov 02 01:54:50.333985 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.333956    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"
Nov 02 01:54:50.349089 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.349063    5875 scope.go:122] "RemoveContainer" containerID="869871651f445395b7e5f1f979bee2a3b2a358a38df394218c5b1d028cd35e41"
Nov 02 01:54:50.349465 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:54:50.349440    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"869871651f445395b7e5f1f979bee2a3b2a358a38df394218c5b1d028cd35e41\": container with ID starting with 869871651f445395b7e5f1f979bee2a3b2a358a38df394218c5b1d028cd35e41 not found: ID does not exist" containerID="869871651f445395b7e5f1f979bee2a3b2a358a38df394218c5b1d028cd35e41"
Nov 02 01:54:50.349545 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.349476    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"869871651f445395b7e5f1f979bee2a3b2a358a38df394218c5b1d028cd35e41"} err="failed to get container status \"869871651f445395b7e5f1f979bee2a3b2a358a38df394218c5b1d028cd35e41\": rpc error: code = NotFound desc = could not find container \"869871651f445395b7e5f1f979bee2a3b2a358a38df394218c5b1d028cd35e41\": container with ID starting with 869871651f445395b7e5f1f979bee2a3b2a358a38df394218c5b1d028cd35e41 not found: ID does not exist"
Nov 02 01:54:50.349545 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.349501    5875 scope.go:122] "RemoveContainer" containerID="940bc71f358809f78adf665b284dcf6feefd4b3c3e6490e71c83c493a11008cf"
Nov 02 01:54:50.362228 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.362199    5875 scope.go:122] "RemoveContainer" containerID="940bc71f358809f78adf665b284dcf6feefd4b3c3e6490e71c83c493a11008cf"
Nov 02 01:54:50.362588 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:54:50.362565    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"940bc71f358809f78adf665b284dcf6feefd4b3c3e6490e71c83c493a11008cf\": container with ID starting with 940bc71f358809f78adf665b284dcf6feefd4b3c3e6490e71c83c493a11008cf not found: ID does not exist" containerID="940bc71f358809f78adf665b284dcf6feefd4b3c3e6490e71c83c493a11008cf"
Nov 02 01:54:50.362672 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.362601    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940bc71f358809f78adf665b284dcf6feefd4b3c3e6490e71c83c493a11008cf"} err="failed to get container status \"940bc71f358809f78adf665b284dcf6feefd4b3c3e6490e71c83c493a11008cf\": rpc error: code = NotFound desc = could not find container \"940bc71f358809f78adf665b284dcf6feefd4b3c3e6490e71c83c493a11008cf\": container with ID starting with 940bc71f358809f78adf665b284dcf6feefd4b3c3e6490e71c83c493a11008cf not found: ID does not exist"
Nov 02 01:54:50.403786 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.403757    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:50.403786 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.403782    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b789m\" (UniqueName: \"kubernetes.io/projected/bafe4c70-f1ef-423f-a94c-552f5545ad6d-kube-api-access-b789m\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:50.403786 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.403793    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:50.403982 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.403801    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:50.403982 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.403811    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:50.403982 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.403820    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pt4w\" (UniqueName: \"kubernetes.io/projected/e1d7b68a-92f5-47c6-bedb-d75802e2232d-kube-api-access-7pt4w\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:50.403982 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.403833    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1d7b68a-92f5-47c6-bedb-d75802e2232d-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:50.403982 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.403846    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bafe4c70-f1ef-423f-a94c-552f5545ad6d-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:50.581954 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.581921    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"]
Nov 02 01:54:50.589351 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.589317    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-gcng9"]
Nov 02 01:54:50.695810 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.695724    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bafe4c70-f1ef-423f-a94c-552f5545ad6d" path="/var/lib/kubelet/pods/bafe4c70-f1ef-423f-a94c-552f5545ad6d/volumes"
Nov 02 01:54:50.701453 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.701429    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"]
Nov 02 01:54:50.707041 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.707020    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-dthb5"]
Nov 02 01:54:50.866877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.866845    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1d5f2ad-mwkk7"]
Nov 02 01:54:50.867487 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.867474    5875 cpu_manager.go:414] "containerMap: removing container" podUID="bafe4c70-f1ef-423f-a94c-552f5545ad6d" containerName="istio-proxy"
Nov 02 01:54:50.867545 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.867489    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bafe4c70-f1ef-423f-a94c-552f5545ad6d" containerName="istio-proxy"
Nov 02 01:54:50.867545 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.867517    5875 cpu_manager.go:414] "containerMap: removing container" podUID="e1d7b68a-92f5-47c6-bedb-d75802e2232d" containerName="istio-proxy"
Nov 02 01:54:50.867545 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.867522    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1d7b68a-92f5-47c6-bedb-d75802e2232d" containerName="istio-proxy"
Nov 02 01:54:50.867545 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.867538    5875 cpu_manager.go:414] "containerMap: removing container" podUID="f0610139-0aec-49ea-a67f-e7815c2f6fe4" containerName="revoke-keys"
Nov 02 01:54:50.867545 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.867543    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0610139-0aec-49ea-a67f-e7815c2f6fe4" containerName="revoke-keys"
Nov 02 01:54:50.867715 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.867649    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1d7b68a-92f5-47c6-bedb-d75802e2232d" containerName="istio-proxy"
Nov 02 01:54:50.867715 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.867659    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0610139-0aec-49ea-a67f-e7815c2f6fe4" containerName="revoke-keys"
Nov 02 01:54:50.867715 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.867670    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="bafe4c70-f1ef-423f-a94c-552f5545ad6d" containerName="istio-proxy"
Nov 02 01:54:50.872504 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.872489    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d5f2ad-mwkk7"
Nov 02 01:54:50.924660 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.924631    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:54:50.924660 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.924638    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:54:50.929085 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:50.929056    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1d5f2ad-mwkk7"]
Nov 02 01:54:51.022566 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.022533    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z66k5\" (UniqueName: \"kubernetes.io/projected/1e532b09-28a0-464c-ad90-33d8aa7b4a0a-kube-api-access-z66k5\") pod \"maas-api-revoke-sub-b1d5f2ad-mwkk7\" (UID: \"1e532b09-28a0-464c-ad90-33d8aa7b4a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d5f2ad-mwkk7"
Nov 02 01:54:51.022736 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.022607    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e532b09-28a0-464c-ad90-33d8aa7b4a0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1d5f2ad-mwkk7\" (UID: \"1e532b09-28a0-464c-ad90-33d8aa7b4a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d5f2ad-mwkk7"
Nov 02 01:54:51.022846 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.022826    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e532b09-28a0-464c-ad90-33d8aa7b4a0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1d5f2ad-mwkk7\" (UID: \"1e532b09-28a0-464c-ad90-33d8aa7b4a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d5f2ad-mwkk7"
Nov 02 01:54:51.124459 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.124421    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e532b09-28a0-464c-ad90-33d8aa7b4a0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1d5f2ad-mwkk7\" (UID: \"1e532b09-28a0-464c-ad90-33d8aa7b4a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d5f2ad-mwkk7"
Nov 02 01:54:51.124608 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.124548    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z66k5\" (UniqueName: \"kubernetes.io/projected/1e532b09-28a0-464c-ad90-33d8aa7b4a0a-kube-api-access-z66k5\") pod \"maas-api-revoke-sub-b1d5f2ad-mwkk7\" (UID: \"1e532b09-28a0-464c-ad90-33d8aa7b4a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d5f2ad-mwkk7"
Nov 02 01:54:51.124653 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.124615    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e532b09-28a0-464c-ad90-33d8aa7b4a0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1d5f2ad-mwkk7\" (UID: \"1e532b09-28a0-464c-ad90-33d8aa7b4a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d5f2ad-mwkk7"
Nov 02 01:54:51.125271 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.125210    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e532b09-28a0-464c-ad90-33d8aa7b4a0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1d5f2ad-mwkk7\" (UID: \"1e532b09-28a0-464c-ad90-33d8aa7b4a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d5f2ad-mwkk7"
Nov 02 01:54:51.126950 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.126929    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e532b09-28a0-464c-ad90-33d8aa7b4a0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1d5f2ad-mwkk7\" (UID: \"1e532b09-28a0-464c-ad90-33d8aa7b4a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d5f2ad-mwkk7"
Nov 02 01:54:51.185586 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.185561    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z66k5\" (UniqueName: \"kubernetes.io/projected/1e532b09-28a0-464c-ad90-33d8aa7b4a0a-kube-api-access-z66k5\") pod \"maas-api-revoke-sub-b1d5f2ad-mwkk7\" (UID: \"1e532b09-28a0-464c-ad90-33d8aa7b4a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d5f2ad-mwkk7"
Nov 02 01:54:51.185883 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.185729    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d5f2ad-mwkk7"
Nov 02 01:54:51.511664 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.511631    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"]
Nov 02 01:54:51.517399 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.517378    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.602956 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.602366    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-15487453-openshift-default-dockercfg-lnpfb\""
Nov 02 01:54:51.644245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.634743    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-podinfo\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.644245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.634799    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-data\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.644245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.634866    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-workload-socket\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.644245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.634889    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-workload-certs\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.644245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.634931    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mh8z\" (UniqueName: \"kubernetes.io/projected/222ac2e9-73cf-4ad3-8704-807adb22fb1a-kube-api-access-6mh8z\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.644245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.634975    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-token\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.644245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.635022    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istiod-ca-cert\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.644245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.635150    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-envoy\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.644245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.635210    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-credential-socket\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.766801 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.766675    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mh8z\" (UniqueName: \"kubernetes.io/projected/222ac2e9-73cf-4ad3-8704-807adb22fb1a-kube-api-access-6mh8z\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.767080 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.767053    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-token\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.767350 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.767268    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istiod-ca-cert\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.767544 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.767525    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-envoy\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.775359 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.775326    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-credential-socket\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.775902 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.775878    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-podinfo\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.776080 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.776061    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-data\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.776295 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.776275    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-workload-socket\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.782273 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.776445    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-workload-certs\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.794631 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.780583    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-workload-certs\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.794853 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.790308    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-credential-socket\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.794990 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.790840    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-data\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.795844 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.795817    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istiod-ca-cert\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.806097 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.806044    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-workload-socket\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.889943 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.889667    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1d5f2ad-mwkk7"]
Nov 02 01:54:51.897865 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.897829    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"]
Nov 02 01:54:51.903743 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.903665    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-token\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.906739 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.906678    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-podinfo\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.907777 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.907754    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mh8z\" (UniqueName: \"kubernetes.io/projected/222ac2e9-73cf-4ad3-8704-807adb22fb1a-kube-api-access-6mh8z\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.908770 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.908678    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-envoy\") pod \"e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") " pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:51.973561 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:51.973033    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:52.353683 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:52.353649    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d5f2ad-mwkk7" event={"ID":"1e532b09-28a0-464c-ad90-33d8aa7b4a0a","Type":"ContainerStarted","Data":"3ebce0bf1c16d93e6c47fce1be93fe77531133a468d59dcdeb9ad746dc478a8e"}
Nov 02 01:54:52.354048 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:52.353708    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d5f2ad-mwkk7" event={"ID":"1e532b09-28a0-464c-ad90-33d8aa7b4a0a","Type":"ContainerStarted","Data":"bfea0de3b5b79080ef2dbd970dea00f665ae1c92d0c58bfbee2fbfd48e790884"}
Nov 02 01:54:52.439692 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:52.439660    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"]
Nov 02 01:54:52.467954 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:54:52.467931    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod222ac2e9_73cf_4ad3_8704_807adb22fb1a.slice/crio-f0e4bba7d524db4f471e21f491de269b8c051998b1b16192bc06f79d704de8bd WatchSource:0}: Error finding container f0e4bba7d524db4f471e21f491de269b8c051998b1b16192bc06f79d704de8bd: Status 404 returned error can't find the container with id f0e4bba7d524db4f471e21f491de269b8c051998b1b16192bc06f79d704de8bd
Nov 02 01:54:52.471551 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:52.471513    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:54:52.471645 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:52.471597    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:54:52.471645 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:52.471637    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:54:52.471712 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:52.471665    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d5f2ad-mwkk7" podStartSLOduration=2.471652422 podStartE2EDuration="2.471652422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:52.468462627 +0000 UTC m=+1339.909499139" watchObservedRunningTime="2026-11-02 01:54:52.471652422 +0000 UTC m=+1339.912688956"
Nov 02 01:54:52.707116 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:52.707089    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1d7b68a-92f5-47c6-bedb-d75802e2232d" path="/var/lib/kubelet/pods/e1d7b68a-92f5-47c6-bedb-d75802e2232d/volumes"
Nov 02 01:54:53.363388 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:53.363300    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf" event={"ID":"222ac2e9-73cf-4ad3-8704-807adb22fb1a","Type":"ContainerStarted","Data":"c2373a48e2a426084eb3835bc8e26371b5176832c5f307d58be2d610b8b8df31"}
Nov 02 01:54:53.365890 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:53.363392    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf" event={"ID":"222ac2e9-73cf-4ad3-8704-807adb22fb1a","Type":"ContainerStarted","Data":"f0e4bba7d524db4f471e21f491de269b8c051998b1b16192bc06f79d704de8bd"}
Nov 02 01:54:53.365890 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:53.364924    5875 generic.go:356] "Generic (PLEG): container finished" podID="1e532b09-28a0-464c-ad90-33d8aa7b4a0a" containerID="3ebce0bf1c16d93e6c47fce1be93fe77531133a468d59dcdeb9ad746dc478a8e" exitCode=0
Nov 02 01:54:53.365890 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:53.364967    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d5f2ad-mwkk7" event={"ID":"1e532b09-28a0-464c-ad90-33d8aa7b4a0a","Type":"ContainerDied","Data":"3ebce0bf1c16d93e6c47fce1be93fe77531133a468d59dcdeb9ad746dc478a8e"}
Nov 02 01:54:53.485856 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:53.485805    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf" podStartSLOduration=2.485793551 podStartE2EDuration="2.485793551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:53.485464824 +0000 UTC m=+1340.926501322" watchObservedRunningTime="2026-11-02 01:54:53.485793551 +0000 UTC m=+1340.926830120"
Nov 02 01:54:53.974517 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:53.974480    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:53.980526 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:53.980504    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:54.374134 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:54.374048    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:54.375262 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:54.375239    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:54:54.575076 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:54.575048    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d5f2ad-mwkk7"
Nov 02 01:54:54.580947 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:54.580921    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:54:54.645324 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:54.645243    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e532b09-28a0-464c-ad90-33d8aa7b4a0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e532b09-28a0-464c-ad90-33d8aa7b4a0a-maas-api-xks-ca\") pod \"1e532b09-28a0-464c-ad90-33d8aa7b4a0a\" (UID: \"1e532b09-28a0-464c-ad90-33d8aa7b4a0a\") "
Nov 02 01:54:54.645324 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:54.645299    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e532b09-28a0-464c-ad90-33d8aa7b4a0a-kube-api-access-z66k5\" (UniqueName: \"kubernetes.io/projected/1e532b09-28a0-464c-ad90-33d8aa7b4a0a-kube-api-access-z66k5\") pod \"1e532b09-28a0-464c-ad90-33d8aa7b4a0a\" (UID: \"1e532b09-28a0-464c-ad90-33d8aa7b4a0a\") "
Nov 02 01:54:54.645544 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:54.645337    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e532b09-28a0-464c-ad90-33d8aa7b4a0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e532b09-28a0-464c-ad90-33d8aa7b4a0a-maas-api-openshift-ca\") pod \"1e532b09-28a0-464c-ad90-33d8aa7b4a0a\" (UID: \"1e532b09-28a0-464c-ad90-33d8aa7b4a0a\") "
Nov 02 01:54:54.645767 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:54.645737    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e532b09-28a0-464c-ad90-33d8aa7b4a0a-maas-api-openshift-ca" pod "1e532b09-28a0-464c-ad90-33d8aa7b4a0a" (UID: "1e532b09-28a0-464c-ad90-33d8aa7b4a0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:54.646062 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:54.646034    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e532b09-28a0-464c-ad90-33d8aa7b4a0a-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:54.647707 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:54.647677    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e532b09-28a0-464c-ad90-33d8aa7b4a0a-kube-api-access-z66k5" pod "1e532b09-28a0-464c-ad90-33d8aa7b4a0a" (UID: "1e532b09-28a0-464c-ad90-33d8aa7b4a0a"). InnerVolumeSpecName "kube-api-access-z66k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:54.647808 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:54.647736    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e532b09-28a0-464c-ad90-33d8aa7b4a0a-maas-api-xks-ca" pod "1e532b09-28a0-464c-ad90-33d8aa7b4a0a" (UID: "1e532b09-28a0-464c-ad90-33d8aa7b4a0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:54:54.747589 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:54.747555    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e532b09-28a0-464c-ad90-33d8aa7b4a0a-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:54.747589 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:54.747593    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z66k5\" (UniqueName: \"kubernetes.io/projected/1e532b09-28a0-464c-ad90-33d8aa7b4a0a-kube-api-access-z66k5\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:55.145322 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.145284    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"]
Nov 02 01:54:55.146036 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.146016    5875 cpu_manager.go:414] "containerMap: removing container" podUID="1e532b09-28a0-464c-ad90-33d8aa7b4a0a" containerName="revoke-keys"
Nov 02 01:54:55.146036 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.146033    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e532b09-28a0-464c-ad90-33d8aa7b4a0a" containerName="revoke-keys"
Nov 02 01:54:55.146207 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.146162    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e532b09-28a0-464c-ad90-33d8aa7b4a0a" containerName="revoke-keys"
Nov 02 01:54:55.149895 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.149880    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.158305 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.158284    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istiod-ca-cert\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.158401 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.158324    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-data\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.158493 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.158478    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-envoy\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.158528 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.158510    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-workload-socket\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.158609 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.158597    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-token\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.158644 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.158626    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-podinfo\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.158731 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.158719    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-credential-socket\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.158766 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.158744    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkr4m\" (UniqueName: \"kubernetes.io/projected/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-kube-api-access-fkr4m\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.158830 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.158813    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-workload-certs\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.212562 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.212531    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-495b48c2-openshift-default-dockercfg-2lc5r\""
Nov 02 01:54:55.217027 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.217001    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"]
Nov 02 01:54:55.260049 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.260013    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-podinfo\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.260214 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.260105    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-credential-socket\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.260214 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.260126    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkr4m\" (UniqueName: \"kubernetes.io/projected/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-kube-api-access-fkr4m\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.260214 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.260151    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-workload-certs\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.260214 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.260192    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istiod-ca-cert\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.260443 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.260252    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-data\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.260443 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.260358    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-envoy\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.260443 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.260435    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-workload-socket\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.260583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.260509    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-token\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.260642 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.260616    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-credential-socket\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.260917 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.260863    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-workload-certs\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.260917 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.260891    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-data\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.261326 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.260941    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istiod-ca-cert\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.261326 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.260942    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-workload-socket\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.262926 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.262905    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-envoy\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.263033 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.263006    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-podinfo\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.329342 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.329314    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-token\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.329605 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.329587    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkr4m\" (UniqueName: \"kubernetes.io/projected/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-kube-api-access-fkr4m\") pod \"e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") " pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:55.383141 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.383110    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d5f2ad-mwkk7"
Nov 02 01:54:55.383141 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.383130    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d5f2ad-mwkk7" event={"ID":"1e532b09-28a0-464c-ad90-33d8aa7b4a0a","Type":"ContainerDied","Data":"bfea0de3b5b79080ef2dbd970dea00f665ae1c92d0c58bfbee2fbfd48e790884"}
Nov 02 01:54:55.383617 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.383188    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfea0de3b5b79080ef2dbd970dea00f665ae1c92d0c58bfbee2fbfd48e790884"
Nov 02 01:54:55.463240 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:55.463193    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:56.052259 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:56.042432    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"]
Nov 02 01:54:56.209435 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:56.209395    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:54:56.209646 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:56.209622    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:54:56.209799 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:56.209778    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:54:57.305250 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.304106    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"]
Nov 02 01:54:57.352718 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.352638    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"]
Nov 02 01:54:57.411511 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.411476    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5" event={"ID":"a47b2793-7e4c-4cae-8d55-6bc82c1e4006","Type":"ContainerStarted","Data":"91c9cd61126fea33741b7bac77e75727f20d314e7c65f4753f9b5609fe7085f8"}
Nov 02 01:54:57.411761 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.410316    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-credential-socket\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.412084 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.412057    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-workload-socket\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.412670 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.396466    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.517630 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.517598    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-b9a2fcb6-gw-openshift-default-dockercfg-rbxl5\""
Nov 02 01:54:57.519988 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.519957    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-workload-certs\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.520238 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.520198    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-data\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.520432 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.520398    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-credential-socket\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.521484 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.521450    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-envoy\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.521744 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.521721    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxrk7\" (UniqueName: \"kubernetes.io/projected/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-kube-api-access-gxrk7\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.528513 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.521044    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-credential-socket\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.528793 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.528461    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-workload-socket\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.529209 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.529182    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-token\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.530195 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.530173    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istiod-ca-cert\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.535279 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.529887    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-workload-socket\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.537189 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.537153    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-podinfo\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.649640 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.645171    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-token\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.649971 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.649948    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istiod-ca-cert\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.650993 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.650959    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-podinfo\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.651638 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.650873    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istiod-ca-cert\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.651849 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.651831    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-workload-certs\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.652344 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.652215    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-workload-certs\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.652533 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.652503    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-data\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.656344 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.652879    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-data\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.656583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.656564    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-envoy\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.657108 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.657086    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxrk7\" (UniqueName: \"kubernetes.io/projected/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-kube-api-access-gxrk7\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.675114 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.675084    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-podinfo\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.681532 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.681498    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-envoy\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.830248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.816640    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-token\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:57.875262 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:57.875196    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxrk7\" (UniqueName: \"kubernetes.io/projected/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-kube-api-access-gxrk7\") pod \"e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") " pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:58.075650 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:58.075608    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:54:58.199007 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:58.198379    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:58.259244 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:58.252515    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" podUID="6e39fc3d-d730-4df7-8d10-2376656a88d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:54:58.435322 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:58.433668    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5" event={"ID":"a47b2793-7e4c-4cae-8d55-6bc82c1e4006","Type":"ContainerStarted","Data":"352ba78804bb01a34735761f5821a644d72cfe16add40b61bbb42bcd1c5f0089"}
Nov 02 01:54:58.481249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:58.469486    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:54:58.481249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:58.477347    5875 patch_prober.go:29] interesting pod/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 02 01:54:58.481249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:58.477404    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5" podUID="a47b2793-7e4c-4cae-8d55-6bc82c1e4006" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 02 01:54:58.721400 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:58.684002    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:58.798256 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:58.778236    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb" podUID="c9628178-6287-4ba3-9467-93f84ae883ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:54:59.132368 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:59.128485    5875 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded"
Nov 02 01:54:59.202432 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:59.202372    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:59.225391 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:59.225341    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:54:59.225568 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:59.225403    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt" podUID="3c4cedf1-b41a-42b5-9b5f-880b8264d434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:59.225665 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:59.225647    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:54:59.257656 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:59.257573    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" podUID="6e39fc3d-d730-4df7-8d10-2376656a88d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:59.288450 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:59.288347    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:54:59.416718 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:59.416680    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:54:59.421353 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:59.421303    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh" podUID="05f7e0b6-dc17-4891-9817-242be8465c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:59.532408 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:59.532340    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5" podStartSLOduration=4.532316603 podStartE2EDuration="4.532316603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:58.696505725 +0000 UTC m=+1346.137542223" watchObservedRunningTime="2026-11-02 01:54:59.532316603 +0000 UTC m=+1346.973353103"
Nov 02 01:54:59.582686 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:59.582592    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:54:59.582973 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:59.582927    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g" podUID="0c95a482-2bd1-457e-9f4a-facf829b7699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:59.709481 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:59.709444    5875 patch_prober.go:29] interesting pod/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 02 01:54:59.713248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:59.709696    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5" podUID="a47b2793-7e4c-4cae-8d55-6bc82c1e4006" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 02 01:54:59.775249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:59.755117    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:59.815309 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:59.777929    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:54:59.839522 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:59.819525    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb" podUID="c9628178-6287-4ba3-9467-93f84ae883ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:54:59.871373 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:59.871338    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:54:59.969486 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:54:59.943912    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"]
Nov 02 01:55:00.012918 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:00.002375    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:55:00.012918 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:00.002473    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:55:00.012918 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:00.002523    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:55:00.231472 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:00.231427    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:00.263560 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:00.263515    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" podUID="6e39fc3d-d730-4df7-8d10-2376656a88d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:55:00.827248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:00.793054    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:01.418620 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:01.384206    5875 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded"
Nov 02 01:55:01.418620 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:01.403697    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb" podUID="c9628178-6287-4ba3-9467-93f84ae883ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:55:01.418620 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:01.417822    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:01.520480 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:01.512100    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:01.520480 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:01.512172    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7" podUID="3726ebef-c581-4736-b0a3-ae417a6e8414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:01.765153 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:01.762651    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" podUID="6e39fc3d-d730-4df7-8d10-2376656a88d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:55:01.889299 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:01.888503    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 01:55:01.930402 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:01.890864    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:01.930402 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:01.890915    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw" podUID="bceaf892-6e4e-4472-bf8d-34b1fe161036" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:02.231320 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:55:02.220346    5875 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.426s"
Nov 02 01:55:02.231320 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:02.220386    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf" event={"ID":"013af201-2be2-4d4a-b07f-edbfd3ec1ac5","Type":"ContainerStarted","Data":"b01011e27dab90bf0aa357718ce40217b772ca1d32bb232d32ee2d1a1dffe666"}
Nov 02 01:55:02.231778 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:02.231716    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:55:02.232012 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:02.231997    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:55:02.259923 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:02.259891    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:55:02.464246 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:02.441376    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:02.508758 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:02.508649    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb" podUID="c9628178-6287-4ba3-9467-93f84ae883ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:02.760308 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:02.752395    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:02.830244 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:02.800317    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b" podUID="7bef4b3e-a32d-4171-bd35-d19e9feef1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:02.845158 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:02.803473    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" podUID="6e39fc3d-d730-4df7-8d10-2376656a88d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:55:02.930247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:02.904525    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 01:55:02.995613 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:02.995565    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:02.995921 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:02.995873    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph" podUID="fb26c061-10c7-4d0c-896c-1fbd27ed00aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:03.265797 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:03.265712    5875 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:03.266067 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:03.266040    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk" podUID="84804eea-7177-41c0-9bd1-63588f418caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:03.483244 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:03.482535    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 01:55:03.690122 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:03.690017    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:55:04.131131 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:04.131089    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf" event={"ID":"013af201-2be2-4d4a-b07f-edbfd3ec1ac5","Type":"ContainerStarted","Data":"f8ab5f234644ab5b25f9760f9c8c0c228d4d09595ea95ff6cd8ef737e7e31fa5"}
Nov 02 01:55:04.168988 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:04.168445    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:55:04.177624 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:04.177600    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:55:04.196838 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:04.196809    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:55:04.537887 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:04.537836    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf" podStartSLOduration=8.537821075 podStartE2EDuration="8.537821075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:04.532366642 +0000 UTC m=+1351.973403140" watchObservedRunningTime="2026-11-02 01:55:04.537821075 +0000 UTC m=+1351.978857572"
Nov 02 01:55:05.076802 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:05.076775    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:55:05.082530 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:05.082507    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:55:05.140254 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:05.140194    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:55:05.140987 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:05.140972    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:55:11.419495 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:55:11.419353    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:55:15.011842 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:15.011633    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q"]
Nov 02 01:55:15.110951 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:15.110924    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q"]
Nov 02 01:55:15.111317 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:15.111255    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q"
Nov 02 01:55:15.169349 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:15.169311    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj7fm\" (UniqueName: \"kubernetes.io/projected/01885305-d407-4274-8c77-a393551d40cd-kube-api-access-hj7fm\") pod \"payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q\" (UID: \"01885305-d407-4274-8c77-a393551d40cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q"
Nov 02 01:55:15.169596 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:15.169446    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01885305-d407-4274-8c77-a393551d40cd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q\" (UID: \"01885305-d407-4274-8c77-a393551d40cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q"
Nov 02 01:55:15.175462 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:15.175435    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-b9a2fcb6\""
Nov 02 01:55:15.217492 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:15.217461    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-b9a2fcb6-dockercfg-74fq5\""
Nov 02 01:55:15.270937 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:15.270808    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj7fm\" (UniqueName: \"kubernetes.io/projected/01885305-d407-4274-8c77-a393551d40cd-kube-api-access-hj7fm\") pod \"payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q\" (UID: \"01885305-d407-4274-8c77-a393551d40cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q"
Nov 02 01:55:15.270937 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:15.270929    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01885305-d407-4274-8c77-a393551d40cd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q\" (UID: \"01885305-d407-4274-8c77-a393551d40cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q"
Nov 02 01:55:15.271660 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:15.271640    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01885305-d407-4274-8c77-a393551d40cd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q\" (UID: \"01885305-d407-4274-8c77-a393551d40cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q"
Nov 02 01:55:15.332304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:15.332274    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj7fm\" (UniqueName: \"kubernetes.io/projected/01885305-d407-4274-8c77-a393551d40cd-kube-api-access-hj7fm\") pod \"payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q\" (UID: \"01885305-d407-4274-8c77-a393551d40cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q"
Nov 02 01:55:15.446207 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:15.446169    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q"
Nov 02 01:55:15.828998 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:15.828968    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q"]
Nov 02 01:55:15.829720 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:55:15.829689    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01885305_d407_4274_8c77_a393551d40cd.slice/crio-578dc101b3760b0bcad3b560e8e16893ed564865137d891ded5d33fe1f74bf63 WatchSource:0}: Error finding container 578dc101b3760b0bcad3b560e8e16893ed564865137d891ded5d33fe1f74bf63: Status 404 returned error can't find the container with id 578dc101b3760b0bcad3b560e8e16893ed564865137d891ded5d33fe1f74bf63
Nov 02 01:55:16.229881 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:16.229834    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q" event={"ID":"01885305-d407-4274-8c77-a393551d40cd","Type":"ContainerStarted","Data":"c8fdaa36d00bd020581ca904b79d71c0a3f56213caf171f36143c802c286de56"}
Nov 02 01:55:16.229881 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:16.229887    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q" event={"ID":"01885305-d407-4274-8c77-a393551d40cd","Type":"ContainerStarted","Data":"578dc101b3760b0bcad3b560e8e16893ed564865137d891ded5d33fe1f74bf63"}
Nov 02 01:55:16.230448 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:16.229991    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q"
Nov 02 01:55:23.970348 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:23.970294    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q" podStartSLOduration=9.970280833 podStartE2EDuration="9.970280833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:16.351746935 +0000 UTC m=+1363.792783432" watchObservedRunningTime="2026-11-02 01:55:23.970280833 +0000 UTC m=+1371.411317337"
Nov 02 01:55:23.972293 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:23.972274    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"]
Nov 02 01:55:23.976985 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:23.976968    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.021167 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.021129    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-envoy\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.021464 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.021440    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-data\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.021626 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.021586    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-workload-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.021973 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.021682    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-token\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.021973 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.021756    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4aa706-493e-42ec-a124-187020d8aa3e-istiod-ca-cert\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.021973 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.021801    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f668\" (UniqueName: \"kubernetes.io/projected/3f4aa706-493e-42ec-a124-187020d8aa3e-kube-api-access-7f668\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.021973 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.021870    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-credential-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.021973 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.021920    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-podinfo\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.022386 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.021996    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-workload-certs\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.026690 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.026663    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"]
Nov 02 01:55:24.032913 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.032892    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.074122 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.074096    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"]
Nov 02 01:55:24.105957 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.104731    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"]
Nov 02 01:55:24.123103 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.123063    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4aa706-493e-42ec-a124-187020d8aa3e-istiod-ca-cert\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.123354 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.123137    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b0c4e37-e032-4b48-8261-81294adf8635-istiod-ca-cert\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.123354 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.123170    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f668\" (UniqueName: \"kubernetes.io/projected/3f4aa706-493e-42ec-a124-187020d8aa3e-kube-api-access-7f668\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.123354 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.123209    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-istio-data\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.123354 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.123314    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-credential-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.123612 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.123376    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-podinfo\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.123612 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.123402    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-workload-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.123612 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.123483    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b0c4e37-e032-4b48-8261-81294adf8635-istio-podinfo\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.123612 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.123516    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-workload-certs\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.123612 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.123542    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktzqf\" (UniqueName: \"kubernetes.io/projected/6b0c4e37-e032-4b48-8261-81294adf8635-kube-api-access-ktzqf\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.123867 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.123645    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-envoy\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.123867 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.123693    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-data\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.123867 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.123740    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-workload-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.123867 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.123830    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-workload-certs\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.123867 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.123863    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-istio-envoy\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.124136 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.123894    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4aa706-493e-42ec-a124-187020d8aa3e-istiod-ca-cert\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.124136 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.123900    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-token\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.124136 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.124021    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b0c4e37-e032-4b48-8261-81294adf8635-istio-token\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.124136 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.124065    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-credential-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.124517 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.124494    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-workload-certs\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.125308 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.124774    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-credential-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.125308 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.125005    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-workload-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.125308 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.125113    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-data\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.127768 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.127741    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-envoy\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.129916 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.129712    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-podinfo\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.184710 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.184673    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-token\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.185232 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.185190    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f668\" (UniqueName: \"kubernetes.io/projected/3f4aa706-493e-42ec-a124-187020d8aa3e-kube-api-access-7f668\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.225049 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.224959    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b0c4e37-e032-4b48-8261-81294adf8635-istio-podinfo\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.225049 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.225010    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktzqf\" (UniqueName: \"kubernetes.io/projected/6b0c4e37-e032-4b48-8261-81294adf8635-kube-api-access-ktzqf\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.225257 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.225148    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-workload-certs\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.225257 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.225169    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-istio-envoy\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.225360 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.225255    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b0c4e37-e032-4b48-8261-81294adf8635-istio-token\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.225360 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.225311    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-credential-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.225457 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.225365    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b0c4e37-e032-4b48-8261-81294adf8635-istiod-ca-cert\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.225457 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.225404    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-istio-data\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.225573 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.225497    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-workload-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.225634 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.225586    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-workload-certs\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.225694 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.225661    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-credential-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.226035 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.226003    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-workload-socket\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.226158 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.226049    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-istio-data\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.226322 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.226300    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b0c4e37-e032-4b48-8261-81294adf8635-istiod-ca-cert\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.227730 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.227707    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-istio-envoy\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.227856 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.227840    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b0c4e37-e032-4b48-8261-81294adf8635-istio-podinfo\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.285338 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.285305    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b0c4e37-e032-4b48-8261-81294adf8635-istio-token\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.285608 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.285580    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktzqf\" (UniqueName: \"kubernetes.io/projected/6b0c4e37-e032-4b48-8261-81294adf8635-kube-api-access-ktzqf\") pod \"e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") " pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.292660 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.292638    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:24.351648 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.351594    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:24.728524 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.728494    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"]
Nov 02 01:55:24.763701 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:55:24.763676    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4aa706_493e_42ec_a124_187020d8aa3e.slice/crio-de15774b6381e904d48e201dbc290cb1d7a1e29e37655f1aeec6d64dc28f7650 WatchSource:0}: Error finding container de15774b6381e904d48e201dbc290cb1d7a1e29e37655f1aeec6d64dc28f7650: Status 404 returned error can't find the container with id de15774b6381e904d48e201dbc290cb1d7a1e29e37655f1aeec6d64dc28f7650
Nov 02 01:55:24.766064 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.766030    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:55:24.766145 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.766094    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:55:24.766145 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.766126    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:55:24.780730 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:55:24.780702    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b0c4e37_e032_4b48_8261_81294adf8635.slice/crio-cc5bfd73b9c3304f874932781bfef18f7833d026c427aaf84549e89df8c59864 WatchSource:0}: Error finding container cc5bfd73b9c3304f874932781bfef18f7833d026c427aaf84549e89df8c59864: Status 404 returned error can't find the container with id cc5bfd73b9c3304f874932781bfef18f7833d026c427aaf84549e89df8c59864
Nov 02 01:55:24.781438 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.781415    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"]
Nov 02 01:55:24.783371 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.783327    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:55:24.783430 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.783406    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:55:24.783472 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:24.783440    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:55:25.331826 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.331785    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n" event={"ID":"6b0c4e37-e032-4b48-8261-81294adf8635","Type":"ContainerStarted","Data":"9b8d5f60b15c963b43d64fec47ac742c52678a4f5f9df8c5dcb09f64c73ce6a7"}
Nov 02 01:55:25.332400 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.331855    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n" event={"ID":"6b0c4e37-e032-4b48-8261-81294adf8635","Type":"ContainerStarted","Data":"cc5bfd73b9c3304f874932781bfef18f7833d026c427aaf84549e89df8c59864"}
Nov 02 01:55:25.335716 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.335676    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f" event={"ID":"3f4aa706-493e-42ec-a124-187020d8aa3e","Type":"ContainerStarted","Data":"ec384196885d2de31d686f8701f35bad124fcebbd5b54b9b32f356131a18677e"}
Nov 02 01:55:25.335859 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.335734    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f" event={"ID":"3f4aa706-493e-42ec-a124-187020d8aa3e","Type":"ContainerStarted","Data":"de15774b6381e904d48e201dbc290cb1d7a1e29e37655f1aeec6d64dc28f7650"}
Nov 02 01:55:25.364586 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.364560    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:25.369281 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.367598    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 01:55:25.369281 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.367653    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n" podUID="6b0c4e37-e032-4b48-8261-81294adf8635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 01:55:25.455659 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.455598    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n" podStartSLOduration=2.4555781469999998 podStartE2EDuration="2.455578147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:25.453733619 +0000 UTC m=+1372.894770119" watchObservedRunningTime="2026-11-02 01:55:25.455578147 +0000 UTC m=+1372.896614643"
Nov 02 01:55:25.530172 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.530137    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"]
Nov 02 01:55:25.534913 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.534891    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.574021 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.573972    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f" podStartSLOduration=2.573958649 podStartE2EDuration="2.573958649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:25.573325892 +0000 UTC m=+1373.014362394" watchObservedRunningTime="2026-11-02 01:55:25.573958649 +0000 UTC m=+1373.014995146"
Nov 02 01:55:25.599057 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.598980    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"]
Nov 02 01:55:25.633529 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.633497    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c637ee2b-9c6b-489e-8f6b-49112217a03c-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.633696 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.633535    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.633696 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.633647    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44dh4\" (UniqueName: \"kubernetes.io/projected/c637ee2b-9c6b-489e-8f6b-49112217a03c-kube-api-access-44dh4\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.633696 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.633683    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.633866 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.633757    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.633899 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.633874    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.633930 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.633899    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.633971 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.633940    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.634009 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.633988    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.735122 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.735081    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c637ee2b-9c6b-489e-8f6b-49112217a03c-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.735122 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.735121    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.735450 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.735178    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44dh4\" (UniqueName: \"kubernetes.io/projected/c637ee2b-9c6b-489e-8f6b-49112217a03c-kube-api-access-44dh4\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.735450 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.735202    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.735450 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.735317    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.735450 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.735388    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.735450 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.735415    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.735706 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.735483    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.735706 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.735522    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.735820 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.735717    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.735820 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.735763    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.735942 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.735844    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.736044 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.736023    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c637ee2b-9c6b-489e-8f6b-49112217a03c-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.736044 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.736037    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.737827 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.737805    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.737984 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.737964    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.815676 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.811783    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44dh4\" (UniqueName: \"kubernetes.io/projected/c637ee2b-9c6b-489e-8f6b-49112217a03c-kube-api-access-44dh4\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.820304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.820244    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.850102 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.849776    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:25.858666 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.858602    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"]
Nov 02 01:55:25.868721 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.868683    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:25.871838 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.871811    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"]
Nov 02 01:55:25.878133 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.877739    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:25.977609 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.977015    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"]
Nov 02 01:55:25.987270 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:25.987239    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"]
Nov 02 01:55:26.057111 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.057073    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-credential-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.057359 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.057337    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh2nc\" (UniqueName: \"kubernetes.io/projected/1a897476-f6fe-494d-b272-2d2f3fa3b08d-kube-api-access-vh2nc\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.057493 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.057462    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-workload-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.057605 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.057588    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-istio-envoy\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.057759 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.057743    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxclv\" (UniqueName: \"kubernetes.io/projected/81b4f431-6173-4de9-b65c-f804391c02bb-kube-api-access-cxclv\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.057890 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.057877    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-workload-certs\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.058011 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.057995    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81b4f431-6173-4de9-b65c-f804391c02bb-istio-podinfo\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.058159 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.058145    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81b4f431-6173-4de9-b65c-f804391c02bb-istio-token\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.058324 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.058294    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-podinfo\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.058427 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.058414    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-envoy\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.058668 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.058651    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-data\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.058792 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.058779    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-istio-data\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.058941 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.058926    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-workload-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.059090 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.059076    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-token\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.059187 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.059172    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81b4f431-6173-4de9-b65c-f804391c02bb-istiod-ca-cert\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.059353 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.059339    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istiod-ca-cert\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.059492 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.059476    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-workload-certs\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.059619 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.059606    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-credential-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.161690 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.161561    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxclv\" (UniqueName: \"kubernetes.io/projected/81b4f431-6173-4de9-b65c-f804391c02bb-kube-api-access-cxclv\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.161690 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.161629    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-workload-certs\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.161690 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.161675    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81b4f431-6173-4de9-b65c-f804391c02bb-istio-podinfo\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.161981 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.161750    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81b4f431-6173-4de9-b65c-f804391c02bb-istio-token\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.161981 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.161804    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-podinfo\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.161981 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.161830    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-envoy\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.161981 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.161872    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-data\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.161981 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.161909    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-istio-data\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.162261 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.161983    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-workload-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.162261 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.162035    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-token\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.162261 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.162063    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81b4f431-6173-4de9-b65c-f804391c02bb-istiod-ca-cert\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.162261 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.162124    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-workload-certs\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.162261 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.162129    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istiod-ca-cert\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.162261 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.162175    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-workload-certs\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.162261 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.162238    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-credential-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.162617 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.162304    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-credential-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.162617 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.162339    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh2nc\" (UniqueName: \"kubernetes.io/projected/1a897476-f6fe-494d-b272-2d2f3fa3b08d-kube-api-access-vh2nc\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.162617 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.162375    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-workload-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.162617 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.162414    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-istio-envoy\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.162617 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.162456    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-istio-data\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.163447 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.163143    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-workload-certs\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.163447 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.163163    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81b4f431-6173-4de9-b65c-f804391c02bb-istiod-ca-cert\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.163447 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.163163    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istiod-ca-cert\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.163447 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.163247    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-workload-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.163447 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.163397    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-data\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.163775 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.163692    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-credential-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.163953 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.163928    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-credential-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.164167 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.164127    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-workload-socket\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.165823 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.165798    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81b4f431-6173-4de9-b65c-f804391c02bb-istio-podinfo\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.166188 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.166130    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-istio-envoy\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.166387 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.166294    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-envoy\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.166814 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.166791    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-podinfo\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.243503 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.243425    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxclv\" (UniqueName: \"kubernetes.io/projected/81b4f431-6173-4de9-b65c-f804391c02bb-kube-api-access-cxclv\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.255410 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.255376    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81b4f431-6173-4de9-b65c-f804391c02bb-istio-token\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:26.255970 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.255900    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-token\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.256198 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.256179    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh2nc\" (UniqueName: \"kubernetes.io/projected/1a897476-f6fe-494d-b272-2d2f3fa3b08d-kube-api-access-vh2nc\") pod \"e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") " pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.294120 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.294091    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:26.295788 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.295762    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 02 01:55:26.295887 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.295805    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f" podUID="3f4aa706-493e-42ec-a124-187020d8aa3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 02 01:55:26.352745 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.352700    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 01:55:26.353196 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.352763    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n" podUID="6b0c4e37-e032-4b48-8261-81294adf8635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 01:55:26.379302 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:55:26.379277    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc637ee2b_9c6b_489e_8f6b_49112217a03c.slice/crio-542d5ad915a61eb010087dc94e940f4a45c2a8c7989d8371d28225511e5b6f76 WatchSource:0}: Error finding container 542d5ad915a61eb010087dc94e940f4a45c2a8c7989d8371d28225511e5b6f76: Status 404 returned error can't find the container with id 542d5ad915a61eb010087dc94e940f4a45c2a8c7989d8371d28225511e5b6f76
Nov 02 01:55:26.379863 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.379841    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"]
Nov 02 01:55:26.382508 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.382473    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:55:26.382728 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.382707    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:55:26.382853 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.382837    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:55:26.485183 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.485157    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:26.495433 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:26.495406    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:27.151310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:27.150178    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"]
Nov 02 01:55:27.166322 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:27.166159    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"]
Nov 02 01:55:27.171435 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:55:27.171303    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b4f431_6173_4de9_b65c_f804391c02bb.slice/crio-bb3f401d2f18e28ef9b69772cb7e88a7fac6f58277fcdcf8feedba061e1a8453 WatchSource:0}: Error finding container bb3f401d2f18e28ef9b69772cb7e88a7fac6f58277fcdcf8feedba061e1a8453: Status 404 returned error can't find the container with id bb3f401d2f18e28ef9b69772cb7e88a7fac6f58277fcdcf8feedba061e1a8453
Nov 02 01:55:27.174319 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:27.173650    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:55:27.174319 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:27.173745    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:55:27.174319 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:27.173802    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:55:27.174319 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:27.173901    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:55:27.174319 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:27.173969    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:55:27.174319 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:27.173998    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:55:27.261126 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:27.259908    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q"
Nov 02 01:55:27.292993 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:27.292973    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 02 01:55:27.293138 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:27.293120    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f" podUID="3f4aa706-493e-42ec-a124-187020d8aa3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 02 01:55:27.366865 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:27.356090    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 01:55:27.366865 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:27.356156    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n" podUID="6b0c4e37-e032-4b48-8261-81294adf8635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 01:55:27.386922 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:27.386677    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2" event={"ID":"1a897476-f6fe-494d-b272-2d2f3fa3b08d","Type":"ContainerStarted","Data":"edd62d3f57de5ff1a3d49db0c9dbc9e511882d32ada0ea9bde50a3dbe9919059"}
Nov 02 01:55:27.389379 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:27.389343    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt" event={"ID":"c637ee2b-9c6b-489e-8f6b-49112217a03c","Type":"ContainerStarted","Data":"ca1bf69d08ce3a5c2eb3ec7b5bfd9869e3244d66b88d50cd5da80b992579494a"}
Nov 02 01:55:27.389522 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:27.389409    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt" event={"ID":"c637ee2b-9c6b-489e-8f6b-49112217a03c","Type":"ContainerStarted","Data":"542d5ad915a61eb010087dc94e940f4a45c2a8c7989d8371d28225511e5b6f76"}
Nov 02 01:55:27.392737 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:27.392712    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x" event={"ID":"81b4f431-6173-4de9-b65c-f804391c02bb","Type":"ContainerStarted","Data":"bb3f401d2f18e28ef9b69772cb7e88a7fac6f58277fcdcf8feedba061e1a8453"}
Nov 02 01:55:27.850849 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:27.850818    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:27.863849 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:27.863817    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 01:55:27.864127 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:27.864096    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt" podUID="c637ee2b-9c6b-489e-8f6b-49112217a03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 01:55:28.297296 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:28.297265    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 02 01:55:28.297614 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:28.297584    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f" podUID="3f4aa706-493e-42ec-a124-187020d8aa3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 02 01:55:28.360998 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:28.352559    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 01:55:28.360998 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:28.352638    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n" podUID="6b0c4e37-e032-4b48-8261-81294adf8635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 01:55:28.457250 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:28.454352    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x" event={"ID":"81b4f431-6173-4de9-b65c-f804391c02bb","Type":"ContainerStarted","Data":"aa04ba1197dd4fe03bf7dee9829967a78ca88eec5848b5500895c355be0ae760"}
Nov 02 01:55:28.485248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:28.468284    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2" event={"ID":"1a897476-f6fe-494d-b272-2d2f3fa3b08d","Type":"ContainerStarted","Data":"de57036e618880da6adb99d173919eb6aa6149811883ac014cb6542700e6564e"}
Nov 02 01:55:28.485688 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:28.485624    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:28.487974 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:28.487946    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 01:55:28.488143 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:28.488117    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2" podUID="1a897476-f6fe-494d-b272-2d2f3fa3b08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 01:55:28.502928 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:28.502898    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:28.503263 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:28.503237    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:55:28.503457 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:28.503431    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x" podUID="81b4f431-6173-4de9-b65c-f804391c02bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 01:55:28.603262 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:28.603118    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt" podStartSLOduration=3.603098016 podStartE2EDuration="3.603098016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:27.579647929 +0000 UTC m=+1375.020684424" watchObservedRunningTime="2026-11-02 01:55:28.603098016 +0000 UTC m=+1376.044134515"
Nov 02 01:55:28.791942 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:28.791862    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2" podStartSLOduration=3.791840586 podStartE2EDuration="3.791840586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:28.766135641 +0000 UTC m=+1376.207172139" watchObservedRunningTime="2026-11-02 01:55:28.791840586 +0000 UTC m=+1376.232877083"
Nov 02 01:55:28.792337 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:28.792291    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x" podStartSLOduration=3.792281041 podStartE2EDuration="3.792281041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:28.616155421 +0000 UTC m=+1376.057191929" watchObservedRunningTime="2026-11-02 01:55:28.792281041 +0000 UTC m=+1376.233317538"
Nov 02 01:55:28.861880 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:28.861788    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 01:55:28.862122 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:28.862095    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt" podUID="c637ee2b-9c6b-489e-8f6b-49112217a03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 01:55:29.180364 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:29.179326    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" podUID="6e39fc3d-d730-4df7-8d10-2376656a88d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:29.241357 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:29.241308    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:29.315895 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:29.315848    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 02 01:55:29.316212 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:29.316162    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f" podUID="3f4aa706-493e-42ec-a124-187020d8aa3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 02 01:55:29.356727 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:29.356603    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 01:55:29.356727 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:29.356664    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n" podUID="6b0c4e37-e032-4b48-8261-81294adf8635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 01:55:29.443434 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:29.443323    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb" podUID="c9628178-6287-4ba3-9467-93f84ae883ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:29.486270 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:29.486200    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 01:55:29.486869 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:29.486839    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2" podUID="1a897476-f6fe-494d-b272-2d2f3fa3b08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 01:55:29.528247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:29.518727    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:55:29.536393 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:29.536325    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x" podUID="81b4f431-6173-4de9-b65c-f804391c02bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 01:55:29.896644 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:29.896592    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 01:55:29.896920 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:29.896880    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt" podUID="c637ee2b-9c6b-489e-8f6b-49112217a03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 01:55:29.908638 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:29.908599    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:30.179630 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:30.179507    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" podUID="6e39fc3d-d730-4df7-8d10-2376656a88d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:30.181160 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:30.181133    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:55:30.246739 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:30.246695    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:30.248514 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:30.247412    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:55:30.293719 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:30.293582    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 02 01:55:30.293719 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:30.293660    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f" podUID="3f4aa706-493e-42ec-a124-187020d8aa3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 02 01:55:30.363470 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:30.363437    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 01:55:30.378561 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:30.363684    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n" podUID="6b0c4e37-e032-4b48-8261-81294adf8635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 01:55:30.553503 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:30.551285    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:55:30.553503 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:30.551343    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x" podUID="81b4f431-6173-4de9-b65c-f804391c02bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 01:55:30.553503 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:30.551429    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 01:55:30.553503 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:30.551452    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2" podUID="1a897476-f6fe-494d-b272-2d2f3fa3b08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 01:55:30.905299 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:30.892207    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 01:55:30.905299 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:30.892286    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt" podUID="c637ee2b-9c6b-489e-8f6b-49112217a03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 01:55:30.924246 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:30.920196    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:30.953245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:30.940145    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:55:30.953245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:30.945044    5875 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded"
Nov 02 01:55:31.216298 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:31.211429    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" podUID="6e39fc3d-d730-4df7-8d10-2376656a88d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:55:31.303267 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:31.279130    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 01:55:31.337244 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:31.310833    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 02 01:55:31.337244 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:31.310924    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f" podUID="3f4aa706-493e-42ec-a124-187020d8aa3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 02 01:55:31.531305 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:31.495791    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 01:55:31.531305 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:31.495872    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n" podUID="6b0c4e37-e032-4b48-8261-81294adf8635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 01:55:31.531305 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:31.518396    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb" podUID="c9628178-6287-4ba3-9467-93f84ae883ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:55:31.583268 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:31.559429    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 01:55:31.583268 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:31.559488    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2" podUID="1a897476-f6fe-494d-b272-2d2f3fa3b08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 01:55:31.583268 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:31.559560    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:55:31.583268 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:31.559581    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x" podUID="81b4f431-6173-4de9-b65c-f804391c02bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 01:55:31.874572 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:31.874483    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 01:55:31.875196 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:31.875078    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt" podUID="c637ee2b-9c6b-489e-8f6b-49112217a03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 01:55:31.947048 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:31.943508    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 01:55:32.245259 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:32.243425    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" podUID="6e39fc3d-d730-4df7-8d10-2376656a88d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:32.342310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:32.329989    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:32.373248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:32.347534    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 02 01:55:32.373248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:32.347600    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f" podUID="3f4aa706-493e-42ec-a124-187020d8aa3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 02 01:55:32.373248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:32.363602    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 01:55:32.373248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:32.363657    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n" podUID="6b0c4e37-e032-4b48-8261-81294adf8635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 01:55:32.497786 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:32.497688    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:55:32.499579 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:32.499544    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x" podUID="81b4f431-6173-4de9-b65c-f804391c02bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 01:55:32.500289 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:32.498039    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 01:55:32.500443 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:32.500422    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2" podUID="1a897476-f6fe-494d-b272-2d2f3fa3b08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 01:55:32.550254 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:32.547237    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb" podUID="c9628178-6287-4ba3-9467-93f84ae883ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:32.550254 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:32.547631    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:55:32.866308 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:32.756169    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:32.866308 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:32.756259    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k" podUID="066ee18e-0d0d-4dee-a06c-3c6b218e1a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:32.903252 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:32.885407    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 01:55:32.903252 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:32.885470    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt" podUID="c637ee2b-9c6b-489e-8f6b-49112217a03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 01:55:33.254308 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:33.120036    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 01:55:33.254308 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:33.252760    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" podUID="6e39fc3d-d730-4df7-8d10-2376656a88d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:33.350752 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:33.350693    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:33.479257 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:33.476851    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 01:55:33.479257 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:33.476912    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n" podUID="6b0c4e37-e032-4b48-8261-81294adf8635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 01:55:33.496039 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:33.496004    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 02 01:55:33.555466 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:33.555366    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f" podUID="3f4aa706-493e-42ec-a124-187020d8aa3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 02 01:55:33.633269 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:33.602776    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb" podUID="c9628178-6287-4ba3-9467-93f84ae883ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:55:33.680262 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:33.678419    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:55:33.680262 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:33.678478    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x" podUID="81b4f431-6173-4de9-b65c-f804391c02bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 01:55:33.779243 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:33.778469    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 01:55:33.779243 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:33.778542    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2" podUID="1a897476-f6fe-494d-b272-2d2f3fa3b08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 01:55:33.789248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:33.779578    5875 scope.go:122] "RemoveContainer" containerID="2eeae924d0c55f636e854230054e10b6f2545c1e7f9b698f854f17f0000b53b0"
Nov 02 01:55:33.900664 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:33.900564    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 01:55:34.307161 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:34.307105    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt" podUID="c637ee2b-9c6b-489e-8f6b-49112217a03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 01:55:34.308347 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:34.154388    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:34.689659 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:34.347173    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" podUID="6e39fc3d-d730-4df7-8d10-2376656a88d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:55:34.946307 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:34.530545    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 01:55:34.946788 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:34.765129    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 01:55:34.946788 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:34.946425    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n" podUID="6b0c4e37-e032-4b48-8261-81294adf8635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 01:55:34.946788 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:34.876775    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb" podUID="c9628178-6287-4ba3-9467-93f84ae883ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 02 01:55:34.962246 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:34.897060    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:55:34.962246 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:34.947625    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x" podUID="81b4f431-6173-4de9-b65c-f804391c02bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 01:55:34.962246 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:34.897802    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 01:55:34.962246 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:34.947684    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2" podUID="1a897476-f6fe-494d-b272-2d2f3fa3b08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 01:55:34.968426 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:34.966764    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 02 01:55:34.968426 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:34.966889    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f" podUID="3f4aa706-493e-42ec-a124-187020d8aa3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 02 01:55:35.019301 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:35.019173    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 01:55:35.019596 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:35.019567    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt" podUID="c637ee2b-9c6b-489e-8f6b-49112217a03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 01:55:35.367720 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:35.367693    5875 scope.go:122] "RemoveContainer" containerID="363089c5ff35f72ae2a7c72aceaa6f14881fff146bcf2c272ba17d9a3256f01b"
Nov 02 01:55:35.386366 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:35.386316    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4d5r" podUID="368af2aa-46fd-4d17-b1c7-fcb9122e1632" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:35.405248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:35.389190    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn" podUID="61e08deb-d43d-4418-8c8d-414c18fa0916" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded"
Nov 02 01:55:35.415352 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:35.415314    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fxwst" podUID="2d0bed6b-fa05-48dd-8f55-21d4903222c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded"
Nov 02 01:55:35.436246 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:35.430373    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 02 01:55:35.556865 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:35.556823    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f" podUID="3f4aa706-493e-42ec-a124-187020d8aa3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 02 01:55:35.565965 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:55:35.447614    5875 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.553s"
Nov 02 01:55:35.568737 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:35.443517    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 01:55:35.582350 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:35.582310    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n" podUID="6b0c4e37-e032-4b48-8261-81294adf8635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 01:55:35.585793 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:35.507517    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:55:35.586003 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:35.585978    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x" podUID="81b4f431-6173-4de9-b65c-f804391c02bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 01:55:35.586147 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:35.507777    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 01:55:35.586292 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:35.586274    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2" podUID="1a897476-f6fe-494d-b272-2d2f3fa3b08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 01:55:35.695389 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:35.695340    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:35.971863 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:35.971820    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 01:55:35.972549 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:35.972498    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt" podUID="c637ee2b-9c6b-489e-8f6b-49112217a03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 01:55:36.011260 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:35.993872    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:55:36.011260 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:36.004128    5875 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" podUID="6e39fc3d-d730-4df7-8d10-2376656a88d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:36.011534 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:36.011390    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" podUID="6e39fc3d-d730-4df7-8d10-2376656a88d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:55:36.039371 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:36.029293    5875 scope.go:122] "RemoveContainer" containerID="f4520f22c4ea896ea4c27c3df21d555ab9b966d2f2ec61eff80d8afcb7a28c92"
Nov 02 01:55:36.143392 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:36.143346    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb" podUID="c9628178-6287-4ba3-9467-93f84ae883ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:36.324077 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:36.324043    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 02 01:55:36.324240 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:36.324103    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f" podUID="3f4aa706-493e-42ec-a124-187020d8aa3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 02 01:55:36.472612 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:36.472570    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 01:55:36.472811 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:36.472635    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n" podUID="6b0c4e37-e032-4b48-8261-81294adf8635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 01:55:36.486171 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:36.486141    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 01:55:36.486378 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:36.486196    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2" podUID="1a897476-f6fe-494d-b272-2d2f3fa3b08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 01:55:36.491414 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:36.491377    5875 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb" podUID="c9628178-6287-4ba3-9467-93f84ae883ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:36.499625 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:36.499596    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:55:36.527352 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:36.527303    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x" podUID="81b4f431-6173-4de9-b65c-f804391c02bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 01:55:36.528900 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:36.528878    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 01:55:36.713612 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:36.705325    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:36.879923 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:36.879803    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 01:55:36.879923 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:36.879859    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt" podUID="c637ee2b-9c6b-489e-8f6b-49112217a03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 01:55:37.160701 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:37.160656    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb" podUID="c9628178-6287-4ba3-9467-93f84ae883ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:37.440643 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:37.440587    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:37.500664 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:37.500628    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:55:37.514464 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:37.514416    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x" podUID="81b4f431-6173-4de9-b65c-f804391c02bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 01:55:37.514739 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:37.501165    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 01:55:37.514880 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:37.514858    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2" podUID="1a897476-f6fe-494d-b272-2d2f3fa3b08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 01:55:37.746248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:37.745972    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:37.878266 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:37.866846    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 01:55:37.878266 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:37.866900    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt" podUID="c637ee2b-9c6b-489e-8f6b-49112217a03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 01:55:38.161982 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:38.161878    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb" podUID="c9628178-6287-4ba3-9467-93f84ae883ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:38.303248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:38.295972    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:38.303248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:38.296092    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f" podUID="3f4aa706-493e-42ec-a124-187020d8aa3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:38.385242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:38.384363    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:38.385242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:38.384451    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n" podUID="6b0c4e37-e032-4b48-8261-81294adf8635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:38.389244 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:38.388833    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" podUID="6e39fc3d-d730-4df7-8d10-2376656a88d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:38.487156 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:38.487045    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 01:55:38.487156 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:38.487101    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2" podUID="1a897476-f6fe-494d-b272-2d2f3fa3b08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 01:55:38.487626 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:38.487590    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:38.488636 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:38.487953    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:55:38.491122 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:38.490785    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 01:55:38.496367 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:38.496336    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:55:38.496478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:38.496396    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x" podUID="81b4f431-6173-4de9-b65c-f804391c02bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 01:55:38.609647 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:38.609470    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 01:55:38.626360 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:38.616338    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 01:55:38.973345 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:38.973282    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 01:55:38.973649 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:38.973604    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt" podUID="c637ee2b-9c6b-489e-8f6b-49112217a03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 01:55:39.299944 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:39.299849    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:39.356245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:39.355678    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:39.356245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:39.355746    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f" podUID="3f4aa706-493e-42ec-a124-187020d8aa3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:39.356245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:39.356063    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:39.429247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:39.428330    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:39.429247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:39.428404    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n" podUID="6b0c4e37-e032-4b48-8261-81294adf8635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:39.486600 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:39.486563    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 02 01:55:39.486772 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:39.486611    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2" podUID="1a897476-f6fe-494d-b272-2d2f3fa3b08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 02 01:55:39.496623 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:39.496590    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 01:55:39.496723 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:39.496653    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x" podUID="81b4f431-6173-4de9-b65c-f804391c02bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 01:55:39.852623 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:39.852580    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 01:55:39.852827 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:39.852645    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt" podUID="c637ee2b-9c6b-489e-8f6b-49112217a03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 01:55:40.313131 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:40.313100    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:40.486243 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:40.476892    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 01:55:40.486545 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:40.486516    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:40.486723 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:40.486697    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f" podUID="3f4aa706-493e-42ec-a124-187020d8aa3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:40.530086 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:40.530057    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:40.533289 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:40.533268    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 01:55:40.850730 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:40.850699    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:41.322717 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:55:41.322675    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:55:41.506977 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:41.485884    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:41.506977 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:41.503479    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:41.506977 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:41.503536    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2" podUID="1a897476-f6fe-494d-b272-2d2f3fa3b08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:41.506977 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:41.505572    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:41.507415 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:41.507375    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:41.507490 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:41.507429    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x" podUID="81b4f431-6173-4de9-b65c-f804391c02bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:41.927250 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:41.926395    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:41.927250 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:41.926529    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt" podUID="c637ee2b-9c6b-489e-8f6b-49112217a03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:42.509277 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:42.506470    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:42.509277 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:42.508495    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:42.509277 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:42.508596    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x" podUID="81b4f431-6173-4de9-b65c-f804391c02bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:42.509277 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:42.508841    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 01:55:42.546424 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:42.546383    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:55:42.546626 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:42.546444    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2" podUID="1a897476-f6fe-494d-b272-2d2f3fa3b08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:55:42.675080 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:42.675037    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:42.677014 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:42.676985    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 01:55:42.779959 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:42.779877    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:42.781330 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:42.781307    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 01:55:45.335098 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.335059    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"]
Nov 02 01:55:45.341962 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.341939    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.384493 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.384465    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"]
Nov 02 01:55:45.389046 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.389027    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.402259 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.402194    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"]
Nov 02 01:55:45.429714 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.429679    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-58w2j\""
Nov 02 01:55:45.451365 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.451337    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"]
Nov 02 01:55:45.477847 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.477808    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbpkv\" (UniqueName: \"kubernetes.io/projected/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-kube-api-access-nbpkv\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.478016 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.477853    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.478016 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.477933    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.478154 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.478058    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.478154 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.478108    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.478154 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.478128    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.478154 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.478144    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.478404 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.478254    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.478404 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.478278    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.478404 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.478377    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.478561 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.478412    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.478561 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.478475    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.478561 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.478503    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.478561 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.478526    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.478561 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.478553    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.478823 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.478569    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.478823 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.478604    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.478823 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.478670    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpmd9\" (UniqueName: \"kubernetes.io/projected/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-kube-api-access-jpmd9\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.580088 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.580054    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.580292 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.580107    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.580292 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.580216    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.580292 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.580258    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.580292 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.580290    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.580489 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.580351    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.580552 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.580514    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.580552 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.580533    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.580661 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.580617    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.580790 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.580758    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.580927 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.580813    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.580927 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.580818    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.580927 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.580909    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.581116 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.580903    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.581116 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.580945    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.581116 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.580968    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.581116 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.580991    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.581116 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.581041    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.581116 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.581085    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.581116 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.581112    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.581520 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.581172    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpmd9\" (UniqueName: \"kubernetes.io/projected/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-kube-api-access-jpmd9\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.581520 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.581299    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbpkv\" (UniqueName: \"kubernetes.io/projected/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-kube-api-access-nbpkv\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.581520 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.581343    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.581520 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.581353    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.581520 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.581382    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.581774 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.581508    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.581774 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.581601    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.581913 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.581866    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.583145 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.583125    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.583473 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.583454    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.584053 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.584030    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.584128 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.584029    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.640765 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.640698    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.641578 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.641552    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.641701 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.641580    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbpkv\" (UniqueName: \"kubernetes.io/projected/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-kube-api-access-nbpkv\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lc78r\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:45.641838 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.641816    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpmd9\" (UniqueName: \"kubernetes.io/projected/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-kube-api-access-jpmd9\") pod \"maas-default-gateway-openshift-default-778b7b5c47-kqj2b\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.656183 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.656164    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:45.705734 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:45.705700    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:46.068204 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:55:46.068160    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070d0963_3e9f_4e5f_920b_1ae95e0c2e81.slice/crio-4cf8fb954577f7c82b22122b858ae7c924485cfe0a54be7ada0de34eb47e2b1e WatchSource:0}: Error finding container 4cf8fb954577f7c82b22122b858ae7c924485cfe0a54be7ada0de34eb47e2b1e: Status 404 returned error can't find the container with id 4cf8fb954577f7c82b22122b858ae7c924485cfe0a54be7ada0de34eb47e2b1e
Nov 02 01:55:46.085383 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:46.072661    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:55:46.085383 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:46.072752    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:55:46.085383 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:46.072819    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:55:46.097292 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:46.097126    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"]
Nov 02 01:55:46.183402 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:55:46.183361    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cf2233f_a377_4bee_9eb9_5d5be2a7561c.slice/crio-4bd52cd57449dfe3436680158063cbc1b3c015d61c6a4c288c2942c53de19f05 WatchSource:0}: Error finding container 4bd52cd57449dfe3436680158063cbc1b3c015d61c6a4c288c2942c53de19f05: Status 404 returned error can't find the container with id 4bd52cd57449dfe3436680158063cbc1b3c015d61c6a4c288c2942c53de19f05
Nov 02 01:55:46.183522 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:46.183495    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"]
Nov 02 01:55:46.186958 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:46.186919    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:55:46.187077 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:46.187010    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:55:46.187077 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:46.187056    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:55:46.530726 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:46.530682    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r" event={"ID":"7cf2233f-a377-4bee-9eb9-5d5be2a7561c","Type":"ContainerStarted","Data":"0d70e52871213e4df11c56d740d6f3fd039f73a183d9c54233e354ef461da577"}
Nov 02 01:55:46.531177 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:46.530749    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r" event={"ID":"7cf2233f-a377-4bee-9eb9-5d5be2a7561c","Type":"ContainerStarted","Data":"4bd52cd57449dfe3436680158063cbc1b3c015d61c6a4c288c2942c53de19f05"}
Nov 02 01:55:46.532570 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:46.532547    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b" event={"ID":"070d0963-3e9f-4e5f-920b-1ae95e0c2e81","Type":"ContainerStarted","Data":"8ebbdeb439ec450146e441538d4592816c73d72158290fc63944b863c4972775"}
Nov 02 01:55:46.532704 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:46.532586    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b" event={"ID":"070d0963-3e9f-4e5f-920b-1ae95e0c2e81","Type":"ContainerStarted","Data":"4cf8fb954577f7c82b22122b858ae7c924485cfe0a54be7ada0de34eb47e2b1e"}
Nov 02 01:55:46.651327 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:46.651267    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r" podStartSLOduration=1.651208306 podStartE2EDuration="1.651208306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:46.650363025 +0000 UTC m=+1394.091399523" watchObservedRunningTime="2026-11-02 01:55:46.651208306 +0000 UTC m=+1394.092244804"
Nov 02 01:55:46.657155 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:46.657131    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:46.705873 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:46.705838    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:46.771826 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:46.771771    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b" podStartSLOduration=1.7717554679999998 podStartE2EDuration="1.771755468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:46.77014002 +0000 UTC m=+1394.211176518" watchObservedRunningTime="2026-11-02 01:55:46.771755468 +0000 UTC m=+1394.212791965"
Nov 02 01:55:47.657773 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:47.657732    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-kqj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 01:55:47.658292 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:47.657793    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b" podUID="070d0963-3e9f-4e5f-920b-1ae95e0c2e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 01:55:47.748248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:47.729452    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-lc78r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 02 01:55:47.748248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:47.729525    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r" podUID="7cf2233f-a377-4bee-9eb9-5d5be2a7561c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 02 01:55:48.656803 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:48.656773    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-kqj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 01:55:48.656977 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:48.656838    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b" podUID="070d0963-3e9f-4e5f-920b-1ae95e0c2e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 01:55:48.708531 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:48.708498    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-lc78r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 02 01:55:48.709126 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:48.709095    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r" podUID="7cf2233f-a377-4bee-9eb9-5d5be2a7561c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 02 01:55:49.657410 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:49.657373    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-kqj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 01:55:49.657604 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:49.657431    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b" podUID="070d0963-3e9f-4e5f-920b-1ae95e0c2e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 01:55:49.714629 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:49.714594    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-lc78r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 02 01:55:49.715282 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:49.715250    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r" podUID="7cf2233f-a377-4bee-9eb9-5d5be2a7561c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 02 01:55:50.656734 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:50.656696    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-kqj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 01:55:50.656924 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:50.656751    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b" podUID="070d0963-3e9f-4e5f-920b-1ae95e0c2e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 01:55:50.717612 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:50.717577    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-lc78r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 02 01:55:50.718685 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:50.718655    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r" podUID="7cf2233f-a377-4bee-9eb9-5d5be2a7561c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 02 01:55:51.657503 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:51.657469    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-kqj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 01:55:51.657666 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:51.657520    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b" podUID="070d0963-3e9f-4e5f-920b-1ae95e0c2e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 01:55:51.718158 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:51.718122    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-lc78r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 02 01:55:51.718796 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:51.718764    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r" podUID="7cf2233f-a377-4bee-9eb9-5d5be2a7561c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 02 01:55:52.657569 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:52.657537    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-kqj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 01:55:52.657794 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:52.657588    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b" podUID="070d0963-3e9f-4e5f-920b-1ae95e0c2e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 01:55:52.706137 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:52.706098    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-lc78r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 02 01:55:52.706316 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:52.706160    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r" podUID="7cf2233f-a377-4bee-9eb9-5d5be2a7561c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 02 01:55:53.657023 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:53.656992    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-kqj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 01:55:53.657617 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:53.657041    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b" podUID="070d0963-3e9f-4e5f-920b-1ae95e0c2e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 01:55:53.706234 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:53.706181    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-lc78r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 02 01:55:53.706424 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:53.706241    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r" podUID="7cf2233f-a377-4bee-9eb9-5d5be2a7561c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 02 01:55:55.051759 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:55.051723    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:55.052431 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:55.052406    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:55.053394 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:55.053373    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 01:55:55.073487 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:55.073458    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:55.073771 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:55.073751    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:55:55.075361 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:55:55.075341    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 01:56:00.178791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:00.178761    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:56:10.416356 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:10.416320    5875 patch_prober.go:29] interesting pod/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:56:10.416765 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:10.416377    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf" podUID="222ac2e9-73cf-4ad3-8704-807adb22fb1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:56:11.321425 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:56:11.321380    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:56:11.898651 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:11.898619    5875 generic.go:356] "Generic (PLEG): container finished" podID="222ac2e9-73cf-4ad3-8704-807adb22fb1a" containerID="c2373a48e2a426084eb3835bc8e26371b5176832c5f307d58be2d610b8b8df31" exitCode=0
Nov 02 01:56:11.899115 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:11.898691    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf" event={"ID":"222ac2e9-73cf-4ad3-8704-807adb22fb1a","Type":"ContainerDied","Data":"c2373a48e2a426084eb3835bc8e26371b5176832c5f307d58be2d610b8b8df31"}
Nov 02 01:56:11.899190 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:11.899177    5875 scope.go:122] "RemoveContainer" containerID="c2373a48e2a426084eb3835bc8e26371b5176832c5f307d58be2d610b8b8df31"
Nov 02 01:56:11.901015 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:11.900984    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:56:11.901098 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:11.901055    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:56:11.901098 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:11.901085    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:56:12.931083 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:12.931044    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf" event={"ID":"222ac2e9-73cf-4ad3-8704-807adb22fb1a","Type":"ContainerStarted","Data":"749ab0743456c2cc5bb61862d893387d8350b7373b7ef8f572a91b5e10d24310"}
Nov 02 01:56:12.975049 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:12.975025    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:56:12.975265 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:12.975181    5875 patch_prober.go:29] interesting pod/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 02 01:56:12.975364 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:12.975267    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf" podUID="222ac2e9-73cf-4ad3-8704-807adb22fb1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 02 01:56:13.974770 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:13.974741    5875 patch_prober.go:29] interesting pod/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 02 01:56:13.975173 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:13.974790    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf" podUID="222ac2e9-73cf-4ad3-8704-807adb22fb1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 02 01:56:14.975284 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:14.975211    5875 patch_prober.go:29] interesting pod/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 02 01:56:14.975697 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:14.975311    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf" podUID="222ac2e9-73cf-4ad3-8704-807adb22fb1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 02 01:56:15.974620 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:15.974593    5875 patch_prober.go:29] interesting pod/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 02 01:56:15.974801 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:15.974645    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf" podUID="222ac2e9-73cf-4ad3-8704-807adb22fb1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 02 01:56:16.975150 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:16.975116    5875 patch_prober.go:29] interesting pod/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 02 01:56:16.975573 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:16.975176    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf" podUID="222ac2e9-73cf-4ad3-8704-807adb22fb1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 02 01:56:17.974787 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:17.974758    5875 patch_prober.go:29] interesting pod/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 02 01:56:17.974982 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:17.974815    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf" podUID="222ac2e9-73cf-4ad3-8704-807adb22fb1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 02 01:56:18.979004 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:18.978966    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:56:18.979543 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:18.979525    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:56:18.980094 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:18.980069    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:56:34.889056 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:34.876473    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:56:41.361712 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:56:41.361631    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:56:47.163722 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:47.163683    5875 patch_prober.go:29] interesting pod/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 02 01:56:47.164256 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:47.163752    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5" podUID="a47b2793-7e4c-4cae-8d55-6bc82c1e4006" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 02 01:56:47.240025 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:47.239997    5875 generic.go:356] "Generic (PLEG): container finished" podID="013af201-2be2-4d4a-b07f-edbfd3ec1ac5" containerID="f8ab5f234644ab5b25f9760f9c8c0c228d4d09595ea95ff6cd8ef737e7e31fa5" exitCode=0
Nov 02 01:56:47.240212 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:47.240066    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf" event={"ID":"013af201-2be2-4d4a-b07f-edbfd3ec1ac5","Type":"ContainerDied","Data":"f8ab5f234644ab5b25f9760f9c8c0c228d4d09595ea95ff6cd8ef737e7e31fa5"}
Nov 02 01:56:47.240526 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:47.240500    5875 scope.go:122] "RemoveContainer" containerID="f8ab5f234644ab5b25f9760f9c8c0c228d4d09595ea95ff6cd8ef737e7e31fa5"
Nov 02 01:56:47.241937 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:47.241918    5875 generic.go:356] "Generic (PLEG): container finished" podID="a47b2793-7e4c-4cae-8d55-6bc82c1e4006" containerID="352ba78804bb01a34735761f5821a644d72cfe16add40b61bbb42bcd1c5f0089" exitCode=0
Nov 02 01:56:47.242013 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:47.241985    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5" event={"ID":"a47b2793-7e4c-4cae-8d55-6bc82c1e4006","Type":"ContainerDied","Data":"352ba78804bb01a34735761f5821a644d72cfe16add40b61bbb42bcd1c5f0089"}
Nov 02 01:56:47.242288 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:47.242261    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:56:47.242350 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:47.242335    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:56:47.242404 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:47.242370    5875 scope.go:122] "RemoveContainer" containerID="352ba78804bb01a34735761f5821a644d72cfe16add40b61bbb42bcd1c5f0089"
Nov 02 01:56:47.242404 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:47.242377    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:56:47.243931 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:47.243903    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:56:47.244021 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:47.243964    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:56:47.244021 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:47.243996    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:56:48.269264 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:48.269215    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf" event={"ID":"013af201-2be2-4d4a-b07f-edbfd3ec1ac5","Type":"ContainerStarted","Data":"509e5d228c9c4555f90288f5d15fd0cf1abedbdd49b5ca945ad31a334deeffa4"}
Nov 02 01:56:48.272409 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:48.272384    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5" event={"ID":"a47b2793-7e4c-4cae-8d55-6bc82c1e4006","Type":"ContainerStarted","Data":"5df611d78c9c4dd82293ccc0bd56e09e53a8bcb04b8b86b4c157016ada6d36c5"}
Nov 02 01:56:48.463864 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:48.463833    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:56:48.464045 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:48.463929    5875 patch_prober.go:29] interesting pod/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 02 01:56:48.464045 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:48.463966    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5" podUID="a47b2793-7e4c-4cae-8d55-6bc82c1e4006" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 02 01:56:48.956256 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:48.956209    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15487453-s829c"]
Nov 02 01:56:48.959973 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:48.959952    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15487453-s829c"
Nov 02 01:56:49.020244 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:49.019573    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:56:49.020439 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:49.020302    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:56:49.064300 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:49.064258    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15487453-s829c"]
Nov 02 01:56:49.066394 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:49.066363    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5358ac77-8542-4ae0-99e9-4d482e3fca7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-15487453-s829c\" (UID: \"5358ac77-8542-4ae0-99e9-4d482e3fca7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15487453-s829c"
Nov 02 01:56:49.066515 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:49.066463    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv5jv\" (UniqueName: \"kubernetes.io/projected/5358ac77-8542-4ae0-99e9-4d482e3fca7e-kube-api-access-jv5jv\") pod \"maas-api-revoke-keys-e2e-ait-15487453-s829c\" (UID: \"5358ac77-8542-4ae0-99e9-4d482e3fca7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15487453-s829c"
Nov 02 01:56:49.067064 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:49.066543    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5358ac77-8542-4ae0-99e9-4d482e3fca7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-15487453-s829c\" (UID: \"5358ac77-8542-4ae0-99e9-4d482e3fca7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15487453-s829c"
Nov 02 01:56:49.076838 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:49.076809    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:56:49.077607 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:49.077579    5875 patch_prober.go:29] interesting pod/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 01:56:49.077724 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:49.077640    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf" podUID="013af201-2be2-4d4a-b07f-edbfd3ec1ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 01:56:49.172885 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:49.172844    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5358ac77-8542-4ae0-99e9-4d482e3fca7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-15487453-s829c\" (UID: \"5358ac77-8542-4ae0-99e9-4d482e3fca7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15487453-s829c"
Nov 02 01:56:49.173091 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:49.172980    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv5jv\" (UniqueName: \"kubernetes.io/projected/5358ac77-8542-4ae0-99e9-4d482e3fca7e-kube-api-access-jv5jv\") pod \"maas-api-revoke-keys-e2e-ait-15487453-s829c\" (UID: \"5358ac77-8542-4ae0-99e9-4d482e3fca7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15487453-s829c"
Nov 02 01:56:49.173183 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:49.173098    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5358ac77-8542-4ae0-99e9-4d482e3fca7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-15487453-s829c\" (UID: \"5358ac77-8542-4ae0-99e9-4d482e3fca7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15487453-s829c"
Nov 02 01:56:49.173888 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:49.173865    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5358ac77-8542-4ae0-99e9-4d482e3fca7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-15487453-s829c\" (UID: \"5358ac77-8542-4ae0-99e9-4d482e3fca7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15487453-s829c"
Nov 02 01:56:49.177871 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:49.177853    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5358ac77-8542-4ae0-99e9-4d482e3fca7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-15487453-s829c\" (UID: \"5358ac77-8542-4ae0-99e9-4d482e3fca7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15487453-s829c"
Nov 02 01:56:49.254566 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:49.254297    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv5jv\" (UniqueName: \"kubernetes.io/projected/5358ac77-8542-4ae0-99e9-4d482e3fca7e-kube-api-access-jv5jv\") pod \"maas-api-revoke-keys-e2e-ait-15487453-s829c\" (UID: \"5358ac77-8542-4ae0-99e9-4d482e3fca7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15487453-s829c"
Nov 02 01:56:49.274513 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:49.274486    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15487453-s829c"
Nov 02 01:56:49.473144 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:49.473108    5875 patch_prober.go:29] interesting pod/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 02 01:56:49.473345 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:49.473166    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5" podUID="a47b2793-7e4c-4cae-8d55-6bc82c1e4006" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 02 01:56:49.764596 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:49.764562    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15487453-s829c"]
Nov 02 01:56:50.077075 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:50.077047    5875 patch_prober.go:29] interesting pod/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 01:56:50.077259 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:50.077106    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf" podUID="013af201-2be2-4d4a-b07f-edbfd3ec1ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 01:56:50.141258 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:50.139906    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:56:50.291893 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:50.291763    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15487453-s829c" event={"ID":"5358ac77-8542-4ae0-99e9-4d482e3fca7e","Type":"ContainerStarted","Data":"0fee191d3567987927e6442fcbb3c0a572194c9b91eeeabffd35384146eac70a"}
Nov 02 01:56:50.463984 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:50.463948    5875 patch_prober.go:29] interesting pod/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 02 01:56:50.464173 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:50.464001    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5" podUID="a47b2793-7e4c-4cae-8d55-6bc82c1e4006" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 02 01:56:51.077264 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:51.077116    5875 patch_prober.go:29] interesting pod/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 01:56:51.077264 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:51.077169    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf" podUID="013af201-2be2-4d4a-b07f-edbfd3ec1ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 01:56:51.344300 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:51.343841    5875 generic.go:356] "Generic (PLEG): container finished" podID="5358ac77-8542-4ae0-99e9-4d482e3fca7e" containerID="a876823ec2d33fcb50be5b3c888236371e706e83dd4151eea39d0f4e0e8c6317" exitCode=0
Nov 02 01:56:51.344300 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:51.343943    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15487453-s829c" event={"ID":"5358ac77-8542-4ae0-99e9-4d482e3fca7e","Type":"ContainerDied","Data":"a876823ec2d33fcb50be5b3c888236371e706e83dd4151eea39d0f4e0e8c6317"}
Nov 02 01:56:51.463769 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:51.463723    5875 patch_prober.go:29] interesting pod/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 02 01:56:51.463930 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:51.463783    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5" podUID="a47b2793-7e4c-4cae-8d55-6bc82c1e4006" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 02 01:56:52.077077 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:52.077043    5875 patch_prober.go:29] interesting pod/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 01:56:52.077280 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:52.077095    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf" podUID="013af201-2be2-4d4a-b07f-edbfd3ec1ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 01:56:52.463837 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:52.463800    5875 patch_prober.go:29] interesting pod/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 02 01:56:52.464384 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:52.463858    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5" podUID="a47b2793-7e4c-4cae-8d55-6bc82c1e4006" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 02 01:56:53.030269 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.023583    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f30cd23a-b8fj2"]
Nov 02 01:56:53.085248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.080824    5875 patch_prober.go:29] interesting pod/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 01:56:53.085248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.080891    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf" podUID="013af201-2be2-4d4a-b07f-edbfd3ec1ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 01:56:53.099260 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.093668    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f30cd23a-b8fj2"]
Nov 02 01:56:53.099260 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.093798    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cd23a-b8fj2"
Nov 02 01:56:53.171641 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.171606    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e898b37-cbe5-427e-9c6c-563dedfc662e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f30cd23a-b8fj2\" (UID: \"7e898b37-cbe5-427e-9c6c-563dedfc662e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cd23a-b8fj2"
Nov 02 01:56:53.171849 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.171829    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtwb9\" (UniqueName: \"kubernetes.io/projected/7e898b37-cbe5-427e-9c6c-563dedfc662e-kube-api-access-mtwb9\") pod \"maas-api-revoke-sub-f30cd23a-b8fj2\" (UID: \"7e898b37-cbe5-427e-9c6c-563dedfc662e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cd23a-b8fj2"
Nov 02 01:56:53.172421 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.172024    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e898b37-cbe5-427e-9c6c-563dedfc662e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f30cd23a-b8fj2\" (UID: \"7e898b37-cbe5-427e-9c6c-563dedfc662e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cd23a-b8fj2"
Nov 02 01:56:53.264994 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.264970    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15487453-s829c"
Nov 02 01:56:53.273526 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.273501    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e898b37-cbe5-427e-9c6c-563dedfc662e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f30cd23a-b8fj2\" (UID: \"7e898b37-cbe5-427e-9c6c-563dedfc662e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cd23a-b8fj2"
Nov 02 01:56:53.273760 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.273745    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtwb9\" (UniqueName: \"kubernetes.io/projected/7e898b37-cbe5-427e-9c6c-563dedfc662e-kube-api-access-mtwb9\") pod \"maas-api-revoke-sub-f30cd23a-b8fj2\" (UID: \"7e898b37-cbe5-427e-9c6c-563dedfc662e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cd23a-b8fj2"
Nov 02 01:56:53.274043 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.274025    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e898b37-cbe5-427e-9c6c-563dedfc662e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f30cd23a-b8fj2\" (UID: \"7e898b37-cbe5-427e-9c6c-563dedfc662e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cd23a-b8fj2"
Nov 02 01:56:53.274097 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.274086    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e898b37-cbe5-427e-9c6c-563dedfc662e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f30cd23a-b8fj2\" (UID: \"7e898b37-cbe5-427e-9c6c-563dedfc662e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cd23a-b8fj2"
Nov 02 01:56:53.276550 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.276528    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e898b37-cbe5-427e-9c6c-563dedfc662e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f30cd23a-b8fj2\" (UID: \"7e898b37-cbe5-427e-9c6c-563dedfc662e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cd23a-b8fj2"
Nov 02 01:56:53.354334 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.354166    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtwb9\" (UniqueName: \"kubernetes.io/projected/7e898b37-cbe5-427e-9c6c-563dedfc662e-kube-api-access-mtwb9\") pod \"maas-api-revoke-sub-f30cd23a-b8fj2\" (UID: \"7e898b37-cbe5-427e-9c6c-563dedfc662e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cd23a-b8fj2"
Nov 02 01:56:53.374739 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.374712    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5358ac77-8542-4ae0-99e9-4d482e3fca7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5358ac77-8542-4ae0-99e9-4d482e3fca7e-maas-api-xks-ca\") pod \"5358ac77-8542-4ae0-99e9-4d482e3fca7e\" (UID: \"5358ac77-8542-4ae0-99e9-4d482e3fca7e\") "
Nov 02 01:56:53.374892 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.374780    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5358ac77-8542-4ae0-99e9-4d482e3fca7e-kube-api-access-jv5jv\" (UniqueName: \"kubernetes.io/projected/5358ac77-8542-4ae0-99e9-4d482e3fca7e-kube-api-access-jv5jv\") pod \"5358ac77-8542-4ae0-99e9-4d482e3fca7e\" (UID: \"5358ac77-8542-4ae0-99e9-4d482e3fca7e\") "
Nov 02 01:56:53.374892 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.374851    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5358ac77-8542-4ae0-99e9-4d482e3fca7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5358ac77-8542-4ae0-99e9-4d482e3fca7e-maas-api-openshift-ca\") pod \"5358ac77-8542-4ae0-99e9-4d482e3fca7e\" (UID: \"5358ac77-8542-4ae0-99e9-4d482e3fca7e\") "
Nov 02 01:56:53.375395 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.375372    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5358ac77-8542-4ae0-99e9-4d482e3fca7e-maas-api-openshift-ca" pod "5358ac77-8542-4ae0-99e9-4d482e3fca7e" (UID: "5358ac77-8542-4ae0-99e9-4d482e3fca7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:56:53.377536 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.377513    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15487453-s829c"
Nov 02 01:56:53.377669 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.377500    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15487453-s829c" event={"ID":"5358ac77-8542-4ae0-99e9-4d482e3fca7e","Type":"ContainerDied","Data":"0fee191d3567987927e6442fcbb3c0a572194c9b91eeeabffd35384146eac70a"}
Nov 02 01:56:53.377669 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.377582    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fee191d3567987927e6442fcbb3c0a572194c9b91eeeabffd35384146eac70a"
Nov 02 01:56:53.377776 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.377712    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5358ac77-8542-4ae0-99e9-4d482e3fca7e-maas-api-xks-ca" pod "5358ac77-8542-4ae0-99e9-4d482e3fca7e" (UID: "5358ac77-8542-4ae0-99e9-4d482e3fca7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:56:53.377776 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.377717    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5358ac77-8542-4ae0-99e9-4d482e3fca7e-kube-api-access-jv5jv" pod "5358ac77-8542-4ae0-99e9-4d482e3fca7e" (UID: "5358ac77-8542-4ae0-99e9-4d482e3fca7e"). InnerVolumeSpecName "kube-api-access-jv5jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:56:53.464032 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.464000    5875 patch_prober.go:29] interesting pod/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 02 01:56:53.464595 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.464051    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5" podUID="a47b2793-7e4c-4cae-8d55-6bc82c1e4006" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 02 01:56:53.477317 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.477283    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5358ac77-8542-4ae0-99e9-4d482e3fca7e-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:53.477467 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.477321    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv5jv\" (UniqueName: \"kubernetes.io/projected/5358ac77-8542-4ae0-99e9-4d482e3fca7e-kube-api-access-jv5jv\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:53.477467 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.477339    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5358ac77-8542-4ae0-99e9-4d482e3fca7e-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:53.563761 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:53.563733    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cd23a-b8fj2"
Nov 02 01:56:54.048332 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:54.048304    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f30cd23a-b8fj2"]
Nov 02 01:56:54.127964 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:54.127938    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:56:54.129300 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:54.129273    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:56:54.390055 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:54.390012    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cd23a-b8fj2" event={"ID":"7e898b37-cbe5-427e-9c6c-563dedfc662e","Type":"ContainerStarted","Data":"09ca40a59fc0220bff6a4a391d0fce65a14fff5aab950b63b940700f5551d876"}
Nov 02 01:56:54.470997 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:54.470959    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:56:54.471468 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:54.471450    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:56:54.472597 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:54.472576    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:56:55.398791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:55.398719    5875 generic.go:356] "Generic (PLEG): container finished" podID="7e898b37-cbe5-427e-9c6c-563dedfc662e" containerID="b2be452aa679989f7fce969b4082e74818e7cbf4ec022e1497a7f4f87250b3c6" exitCode=0
Nov 02 01:56:55.398957 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:55.398832    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cd23a-b8fj2" event={"ID":"7e898b37-cbe5-427e-9c6c-563dedfc662e","Type":"ContainerDied","Data":"b2be452aa679989f7fce969b4082e74818e7cbf4ec022e1497a7f4f87250b3c6"}
Nov 02 01:56:56.564987 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:56.564958    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cd23a-b8fj2"
Nov 02 01:56:56.648845 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:56.648815    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e898b37-cbe5-427e-9c6c-563dedfc662e-kube-api-access-mtwb9\" (UniqueName: \"kubernetes.io/projected/7e898b37-cbe5-427e-9c6c-563dedfc662e-kube-api-access-mtwb9\") pod \"7e898b37-cbe5-427e-9c6c-563dedfc662e\" (UID: \"7e898b37-cbe5-427e-9c6c-563dedfc662e\") "
Nov 02 01:56:56.649005 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:56.648883    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e898b37-cbe5-427e-9c6c-563dedfc662e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e898b37-cbe5-427e-9c6c-563dedfc662e-maas-api-openshift-ca\") pod \"7e898b37-cbe5-427e-9c6c-563dedfc662e\" (UID: \"7e898b37-cbe5-427e-9c6c-563dedfc662e\") "
Nov 02 01:56:56.649005 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:56.648947    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e898b37-cbe5-427e-9c6c-563dedfc662e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e898b37-cbe5-427e-9c6c-563dedfc662e-maas-api-xks-ca\") pod \"7e898b37-cbe5-427e-9c6c-563dedfc662e\" (UID: \"7e898b37-cbe5-427e-9c6c-563dedfc662e\") "
Nov 02 01:56:56.649321 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:56.649294    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e898b37-cbe5-427e-9c6c-563dedfc662e-maas-api-openshift-ca" pod "7e898b37-cbe5-427e-9c6c-563dedfc662e" (UID: "7e898b37-cbe5-427e-9c6c-563dedfc662e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:56:56.649686 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:56.649660    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e898b37-cbe5-427e-9c6c-563dedfc662e-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:56.651190 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:56.651170    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e898b37-cbe5-427e-9c6c-563dedfc662e-maas-api-xks-ca" pod "7e898b37-cbe5-427e-9c6c-563dedfc662e" (UID: "7e898b37-cbe5-427e-9c6c-563dedfc662e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:56:56.651289 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:56.651268    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e898b37-cbe5-427e-9c6c-563dedfc662e-kube-api-access-mtwb9" pod "7e898b37-cbe5-427e-9c6c-563dedfc662e" (UID: "7e898b37-cbe5-427e-9c6c-563dedfc662e"). InnerVolumeSpecName "kube-api-access-mtwb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:56:56.751634 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:56.751603    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtwb9\" (UniqueName: \"kubernetes.io/projected/7e898b37-cbe5-427e-9c6c-563dedfc662e-kube-api-access-mtwb9\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:56.751634 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:56.751635    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e898b37-cbe5-427e-9c6c-563dedfc662e-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:57.415732 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:57.415700    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cd23a-b8fj2"
Nov 02 01:56:57.415914 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:57.415732    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f30cd23a-b8fj2" event={"ID":"7e898b37-cbe5-427e-9c6c-563dedfc662e","Type":"ContainerDied","Data":"09ca40a59fc0220bff6a4a391d0fce65a14fff5aab950b63b940700f5551d876"}
Nov 02 01:56:57.415914 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:57.415778    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09ca40a59fc0220bff6a4a391d0fce65a14fff5aab950b63b940700f5551d876"
Nov 02 01:56:59.638891 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:59.638832    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp"]
Nov 02 01:56:59.639635 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:59.639617    5875 cpu_manager.go:414] "containerMap: removing container" podUID="7e898b37-cbe5-427e-9c6c-563dedfc662e" containerName="revoke-keys"
Nov 02 01:56:59.639682 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:59.639636    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e898b37-cbe5-427e-9c6c-563dedfc662e" containerName="revoke-keys"
Nov 02 01:56:59.639682 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:59.639650    5875 cpu_manager.go:414] "containerMap: removing container" podUID="5358ac77-8542-4ae0-99e9-4d482e3fca7e" containerName="revoke-keys"
Nov 02 01:56:59.639682 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:59.639655    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5358ac77-8542-4ae0-99e9-4d482e3fca7e" containerName="revoke-keys"
Nov 02 01:56:59.639811 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:59.639801    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="5358ac77-8542-4ae0-99e9-4d482e3fca7e" containerName="revoke-keys"
Nov 02 01:56:59.639848 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:59.639825    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e898b37-cbe5-427e-9c6c-563dedfc662e" containerName="revoke-keys"
Nov 02 01:56:59.651605 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:59.651572    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp"
Nov 02 01:56:59.696122 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:59.696080    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3cf00e4-8887-4f12-b0fa-5566ad05e65c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp\" (UID: \"c3cf00e4-8887-4f12-b0fa-5566ad05e65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp"
Nov 02 01:56:59.696359 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:59.696261    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45v22\" (UniqueName: \"kubernetes.io/projected/c3cf00e4-8887-4f12-b0fa-5566ad05e65c-kube-api-access-45v22\") pod \"maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp\" (UID: \"c3cf00e4-8887-4f12-b0fa-5566ad05e65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp"
Nov 02 01:56:59.696497 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:59.696476    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3cf00e4-8887-4f12-b0fa-5566ad05e65c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp\" (UID: \"c3cf00e4-8887-4f12-b0fa-5566ad05e65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp"
Nov 02 01:56:59.701998 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:59.701975    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp"]
Nov 02 01:56:59.744395 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:59.744364    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:56:59.744596 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:59.744532    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:56:59.797853 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:59.797822    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3cf00e4-8887-4f12-b0fa-5566ad05e65c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp\" (UID: \"c3cf00e4-8887-4f12-b0fa-5566ad05e65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp"
Nov 02 01:56:59.798032 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:59.797876    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45v22\" (UniqueName: \"kubernetes.io/projected/c3cf00e4-8887-4f12-b0fa-5566ad05e65c-kube-api-access-45v22\") pod \"maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp\" (UID: \"c3cf00e4-8887-4f12-b0fa-5566ad05e65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp"
Nov 02 01:56:59.798032 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:59.798003    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3cf00e4-8887-4f12-b0fa-5566ad05e65c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp\" (UID: \"c3cf00e4-8887-4f12-b0fa-5566ad05e65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp"
Nov 02 01:56:59.798624 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:59.798597    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3cf00e4-8887-4f12-b0fa-5566ad05e65c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp\" (UID: \"c3cf00e4-8887-4f12-b0fa-5566ad05e65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp"
Nov 02 01:56:59.800617 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:59.800595    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3cf00e4-8887-4f12-b0fa-5566ad05e65c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp\" (UID: \"c3cf00e4-8887-4f12-b0fa-5566ad05e65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp"
Nov 02 01:56:59.859397 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:59.859367    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45v22\" (UniqueName: \"kubernetes.io/projected/c3cf00e4-8887-4f12-b0fa-5566ad05e65c-kube-api-access-45v22\") pod \"maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp\" (UID: \"c3cf00e4-8887-4f12-b0fa-5566ad05e65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp"
Nov 02 01:56:59.965207 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:56:59.965174    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp"
Nov 02 01:57:00.351646 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:57:00.351610    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3cf00e4_8887_4f12_b0fa_5566ad05e65c.slice/crio-4171df41bbaae4fd7a8cd9ecc6d25ff5bf043d1f5bb8e8efb455f2dcac5848a0 WatchSource:0}: Error finding container 4171df41bbaae4fd7a8cd9ecc6d25ff5bf043d1f5bb8e8efb455f2dcac5848a0: Status 404 returned error can't find the container with id 4171df41bbaae4fd7a8cd9ecc6d25ff5bf043d1f5bb8e8efb455f2dcac5848a0
Nov 02 01:57:00.351646 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:00.351631    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp"]
Nov 02 01:57:00.448585 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:00.448550    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp" event={"ID":"c3cf00e4-8887-4f12-b0fa-5566ad05e65c","Type":"ContainerStarted","Data":"4171df41bbaae4fd7a8cd9ecc6d25ff5bf043d1f5bb8e8efb455f2dcac5848a0"}
Nov 02 01:57:01.459670 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:01.459640    5875 generic.go:356] "Generic (PLEG): container finished" podID="c3cf00e4-8887-4f12-b0fa-5566ad05e65c" containerID="89d635edc9dec71bdc170eb17ba2ebca30e58b548feb7bd22273199a068a1f9a" exitCode=0
Nov 02 01:57:01.460072 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:01.459730    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp" event={"ID":"c3cf00e4-8887-4f12-b0fa-5566ad05e65c","Type":"ContainerDied","Data":"89d635edc9dec71bdc170eb17ba2ebca30e58b548feb7bd22273199a068a1f9a"}
Nov 02 01:57:02.633953 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:02.633930    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp"
Nov 02 01:57:02.666406 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:02.666381    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3cf00e4-8887-4f12-b0fa-5566ad05e65c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3cf00e4-8887-4f12-b0fa-5566ad05e65c-maas-api-xks-ca\") pod \"c3cf00e4-8887-4f12-b0fa-5566ad05e65c\" (UID: \"c3cf00e4-8887-4f12-b0fa-5566ad05e65c\") "
Nov 02 01:57:02.666539 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:02.666531    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3cf00e4-8887-4f12-b0fa-5566ad05e65c-kube-api-access-45v22\" (UniqueName: \"kubernetes.io/projected/c3cf00e4-8887-4f12-b0fa-5566ad05e65c-kube-api-access-45v22\") pod \"c3cf00e4-8887-4f12-b0fa-5566ad05e65c\" (UID: \"c3cf00e4-8887-4f12-b0fa-5566ad05e65c\") "
Nov 02 01:57:02.666653 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:02.666638    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3cf00e4-8887-4f12-b0fa-5566ad05e65c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3cf00e4-8887-4f12-b0fa-5566ad05e65c-maas-api-openshift-ca\") pod \"c3cf00e4-8887-4f12-b0fa-5566ad05e65c\" (UID: \"c3cf00e4-8887-4f12-b0fa-5566ad05e65c\") "
Nov 02 01:57:02.667035 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:02.667011    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3cf00e4-8887-4f12-b0fa-5566ad05e65c-maas-api-openshift-ca" pod "c3cf00e4-8887-4f12-b0fa-5566ad05e65c" (UID: "c3cf00e4-8887-4f12-b0fa-5566ad05e65c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:57:02.667329 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:02.667293    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3cf00e4-8887-4f12-b0fa-5566ad05e65c-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:02.668588 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:02.668564    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3cf00e4-8887-4f12-b0fa-5566ad05e65c-maas-api-xks-ca" pod "c3cf00e4-8887-4f12-b0fa-5566ad05e65c" (UID: "c3cf00e4-8887-4f12-b0fa-5566ad05e65c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:57:02.668701 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:02.668686    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3cf00e4-8887-4f12-b0fa-5566ad05e65c-kube-api-access-45v22" pod "c3cf00e4-8887-4f12-b0fa-5566ad05e65c" (UID: "c3cf00e4-8887-4f12-b0fa-5566ad05e65c"). InnerVolumeSpecName "kube-api-access-45v22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:02.769240 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:02.769120    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3cf00e4-8887-4f12-b0fa-5566ad05e65c-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:02.769390 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:02.769255    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45v22\" (UniqueName: \"kubernetes.io/projected/c3cf00e4-8887-4f12-b0fa-5566ad05e65c-kube-api-access-45v22\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:03.528260 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:03.525949    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp" event={"ID":"c3cf00e4-8887-4f12-b0fa-5566ad05e65c","Type":"ContainerDied","Data":"4171df41bbaae4fd7a8cd9ecc6d25ff5bf043d1f5bb8e8efb455f2dcac5848a0"}
Nov 02 01:57:03.528260 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:03.526009    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4171df41bbaae4fd7a8cd9ecc6d25ff5bf043d1f5bb8e8efb455f2dcac5848a0"
Nov 02 01:57:03.528260 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:03.526073    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp"
Nov 02 01:57:09.869593 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:09.869560    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb"]
Nov 02 01:57:09.870261 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:09.870244    5875 cpu_manager.go:414] "containerMap: removing container" podUID="c3cf00e4-8887-4f12-b0fa-5566ad05e65c" containerName="revoke-keys"
Nov 02 01:57:09.870261 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:09.870258    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3cf00e4-8887-4f12-b0fa-5566ad05e65c" containerName="revoke-keys"
Nov 02 01:57:09.870426 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:09.870417    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3cf00e4-8887-4f12-b0fa-5566ad05e65c" containerName="revoke-keys"
Nov 02 01:57:09.873260 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:09.873212    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb"
Nov 02 01:57:09.946378 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:09.946356    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:57:09.958938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:09.958906    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b31c334-b688-4f3a-9d08-d9b2e6ef3d33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb\" (UID: \"4b31c334-b688-4f3a-9d08-d9b2e6ef3d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb"
Nov 02 01:57:09.959089 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:09.959035    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b31c334-b688-4f3a-9d08-d9b2e6ef3d33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb\" (UID: \"4b31c334-b688-4f3a-9d08-d9b2e6ef3d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb"
Nov 02 01:57:09.959273 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:09.959255    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4xl9\" (UniqueName: \"kubernetes.io/projected/4b31c334-b688-4f3a-9d08-d9b2e6ef3d33-kube-api-access-k4xl9\") pod \"maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb\" (UID: \"4b31c334-b688-4f3a-9d08-d9b2e6ef3d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb"
Nov 02 01:57:09.975275 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:09.975254    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:57:09.980409 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:09.980386    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb"]
Nov 02 01:57:10.075257 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:10.073284    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4xl9\" (UniqueName: \"kubernetes.io/projected/4b31c334-b688-4f3a-9d08-d9b2e6ef3d33-kube-api-access-k4xl9\") pod \"maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb\" (UID: \"4b31c334-b688-4f3a-9d08-d9b2e6ef3d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb"
Nov 02 01:57:10.075257 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:10.073390    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b31c334-b688-4f3a-9d08-d9b2e6ef3d33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb\" (UID: \"4b31c334-b688-4f3a-9d08-d9b2e6ef3d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb"
Nov 02 01:57:10.075257 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:10.073492    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b31c334-b688-4f3a-9d08-d9b2e6ef3d33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb\" (UID: \"4b31c334-b688-4f3a-9d08-d9b2e6ef3d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb"
Nov 02 01:57:10.075257 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:10.074874    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b31c334-b688-4f3a-9d08-d9b2e6ef3d33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb\" (UID: \"4b31c334-b688-4f3a-9d08-d9b2e6ef3d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb"
Nov 02 01:57:10.082951 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:10.080950    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b31c334-b688-4f3a-9d08-d9b2e6ef3d33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb\" (UID: \"4b31c334-b688-4f3a-9d08-d9b2e6ef3d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb"
Nov 02 01:57:10.141306 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:10.141210    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4xl9\" (UniqueName: \"kubernetes.io/projected/4b31c334-b688-4f3a-9d08-d9b2e6ef3d33-kube-api-access-k4xl9\") pod \"maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb\" (UID: \"4b31c334-b688-4f3a-9d08-d9b2e6ef3d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb"
Nov 02 01:57:10.185647 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:10.185613    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb"
Nov 02 01:57:10.578152 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:57:10.578121    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b31c334_b688_4f3a_9d08_d9b2e6ef3d33.slice/crio-8c3866e2f16089ad72f997d90ed40a0d497478027954c5dd9d1dc888d5a3415c WatchSource:0}: Error finding container 8c3866e2f16089ad72f997d90ed40a0d497478027954c5dd9d1dc888d5a3415c: Status 404 returned error can't find the container with id 8c3866e2f16089ad72f997d90ed40a0d497478027954c5dd9d1dc888d5a3415c
Nov 02 01:57:10.579085 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:10.579025    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb"]
Nov 02 01:57:11.320267 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:57:11.320137    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:57:11.665864 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:11.665827    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb" event={"ID":"4b31c334-b688-4f3a-9d08-d9b2e6ef3d33","Type":"ContainerStarted","Data":"f899001475e40df82e526f4f7f66c6e3617edb0d191faa7bdadfdb85ecbc1c3d"}
Nov 02 01:57:11.666030 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:11.665883    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb" event={"ID":"4b31c334-b688-4f3a-9d08-d9b2e6ef3d33","Type":"ContainerStarted","Data":"8c3866e2f16089ad72f997d90ed40a0d497478027954c5dd9d1dc888d5a3415c"}
Nov 02 01:57:11.784922 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:11.784868    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb" podStartSLOduration=2.784854469 podStartE2EDuration="2.784854469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:57:11.781846828 +0000 UTC m=+1479.222883325" watchObservedRunningTime="2026-11-02 01:57:11.784854469 +0000 UTC m=+1479.225890966"
Nov 02 01:57:12.848244 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:12.836728    5875 generic.go:356] "Generic (PLEG): container finished" podID="4b31c334-b688-4f3a-9d08-d9b2e6ef3d33" containerID="f899001475e40df82e526f4f7f66c6e3617edb0d191faa7bdadfdb85ecbc1c3d" exitCode=0
Nov 02 01:57:12.942955 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:12.942741    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb" event={"ID":"4b31c334-b688-4f3a-9d08-d9b2e6ef3d33","Type":"ContainerDied","Data":"f899001475e40df82e526f4f7f66c6e3617edb0d191faa7bdadfdb85ecbc1c3d"}
Nov 02 01:57:14.038597 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:14.038575    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb"
Nov 02 01:57:14.121520 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:14.121451    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b31c334-b688-4f3a-9d08-d9b2e6ef3d33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b31c334-b688-4f3a-9d08-d9b2e6ef3d33-maas-api-openshift-ca\") pod \"4b31c334-b688-4f3a-9d08-d9b2e6ef3d33\" (UID: \"4b31c334-b688-4f3a-9d08-d9b2e6ef3d33\") "
Nov 02 01:57:14.121520 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:14.121497    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b31c334-b688-4f3a-9d08-d9b2e6ef3d33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b31c334-b688-4f3a-9d08-d9b2e6ef3d33-maas-api-xks-ca\") pod \"4b31c334-b688-4f3a-9d08-d9b2e6ef3d33\" (UID: \"4b31c334-b688-4f3a-9d08-d9b2e6ef3d33\") "
Nov 02 01:57:14.121728 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:14.121607    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b31c334-b688-4f3a-9d08-d9b2e6ef3d33-kube-api-access-k4xl9\" (UniqueName: \"kubernetes.io/projected/4b31c334-b688-4f3a-9d08-d9b2e6ef3d33-kube-api-access-k4xl9\") pod \"4b31c334-b688-4f3a-9d08-d9b2e6ef3d33\" (UID: \"4b31c334-b688-4f3a-9d08-d9b2e6ef3d33\") "
Nov 02 01:57:14.121794 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:14.121757    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b31c334-b688-4f3a-9d08-d9b2e6ef3d33-maas-api-openshift-ca" pod "4b31c334-b688-4f3a-9d08-d9b2e6ef3d33" (UID: "4b31c334-b688-4f3a-9d08-d9b2e6ef3d33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:57:14.122126 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:14.122106    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b31c334-b688-4f3a-9d08-d9b2e6ef3d33-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:14.123652 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:14.123624    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b31c334-b688-4f3a-9d08-d9b2e6ef3d33-maas-api-xks-ca" pod "4b31c334-b688-4f3a-9d08-d9b2e6ef3d33" (UID: "4b31c334-b688-4f3a-9d08-d9b2e6ef3d33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:57:14.123728 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:14.123686    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b31c334-b688-4f3a-9d08-d9b2e6ef3d33-kube-api-access-k4xl9" pod "4b31c334-b688-4f3a-9d08-d9b2e6ef3d33" (UID: "4b31c334-b688-4f3a-9d08-d9b2e6ef3d33"). InnerVolumeSpecName "kube-api-access-k4xl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:14.223285 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:14.223256    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b31c334-b688-4f3a-9d08-d9b2e6ef3d33-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:14.223285 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:14.223282    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4xl9\" (UniqueName: \"kubernetes.io/projected/4b31c334-b688-4f3a-9d08-d9b2e6ef3d33-kube-api-access-k4xl9\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:14.853203 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:14.853167    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb"
Nov 02 01:57:14.853434 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:14.853231    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb" event={"ID":"4b31c334-b688-4f3a-9d08-d9b2e6ef3d33","Type":"ContainerDied","Data":"8c3866e2f16089ad72f997d90ed40a0d497478027954c5dd9d1dc888d5a3415c"}
Nov 02 01:57:14.853434 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:14.853274    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c3866e2f16089ad72f997d90ed40a0d497478027954c5dd9d1dc888d5a3415c"
Nov 02 01:57:21.423614 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:21.423582    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q"]
Nov 02 01:57:26.424381 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:26.424320    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q" podUID="01885305-d407-4274-8c77-a393551d40cd" containerName="payload-processing" containerID="cri-o://c8fdaa36d00bd020581ca904b79d71c0a3f56213caf171f36143c802c286de56" gracePeriod=25
Nov 02 01:57:26.725275 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:26.725248    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q"
Nov 02 01:57:26.815736 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:26.815705    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01885305-d407-4274-8c77-a393551d40cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01885305-d407-4274-8c77-a393551d40cd-plugins-config-volume\") pod \"01885305-d407-4274-8c77-a393551d40cd\" (UID: \"01885305-d407-4274-8c77-a393551d40cd\") "
Nov 02 01:57:26.815945 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:26.815932    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01885305-d407-4274-8c77-a393551d40cd-kube-api-access-hj7fm\" (UniqueName: \"kubernetes.io/projected/01885305-d407-4274-8c77-a393551d40cd-kube-api-access-hj7fm\") pod \"01885305-d407-4274-8c77-a393551d40cd\" (UID: \"01885305-d407-4274-8c77-a393551d40cd\") "
Nov 02 01:57:26.816046 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:26.816027    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01885305-d407-4274-8c77-a393551d40cd-plugins-config-volume" pod "01885305-d407-4274-8c77-a393551d40cd" (UID: "01885305-d407-4274-8c77-a393551d40cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:57:26.816420 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:26.816402    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01885305-d407-4274-8c77-a393551d40cd-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:26.818082 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:26.818060    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01885305-d407-4274-8c77-a393551d40cd-kube-api-access-hj7fm" pod "01885305-d407-4274-8c77-a393551d40cd" (UID: "01885305-d407-4274-8c77-a393551d40cd"). InnerVolumeSpecName "kube-api-access-hj7fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:26.917871 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:26.917837    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj7fm\" (UniqueName: \"kubernetes.io/projected/01885305-d407-4274-8c77-a393551d40cd-kube-api-access-hj7fm\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:27.093310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:27.093280    5875 generic.go:356] "Generic (PLEG): container finished" podID="01885305-d407-4274-8c77-a393551d40cd" containerID="c8fdaa36d00bd020581ca904b79d71c0a3f56213caf171f36143c802c286de56" exitCode=0
Nov 02 01:57:27.093476 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:27.093328    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q"
Nov 02 01:57:27.093476 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:27.093363    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q" event={"ID":"01885305-d407-4274-8c77-a393551d40cd","Type":"ContainerDied","Data":"c8fdaa36d00bd020581ca904b79d71c0a3f56213caf171f36143c802c286de56"}
Nov 02 01:57:27.093476 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:27.093421    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q" event={"ID":"01885305-d407-4274-8c77-a393551d40cd","Type":"ContainerDied","Data":"578dc101b3760b0bcad3b560e8e16893ed564865137d891ded5d33fe1f74bf63"}
Nov 02 01:57:27.093476 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:27.093450    5875 scope.go:122] "RemoveContainer" containerID="c8fdaa36d00bd020581ca904b79d71c0a3f56213caf171f36143c802c286de56"
Nov 02 01:57:27.104599 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:27.104577    5875 scope.go:122] "RemoveContainer" containerID="c8fdaa36d00bd020581ca904b79d71c0a3f56213caf171f36143c802c286de56"
Nov 02 01:57:27.104863 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:57:27.104844    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8fdaa36d00bd020581ca904b79d71c0a3f56213caf171f36143c802c286de56\": container with ID starting with c8fdaa36d00bd020581ca904b79d71c0a3f56213caf171f36143c802c286de56 not found: ID does not exist" containerID="c8fdaa36d00bd020581ca904b79d71c0a3f56213caf171f36143c802c286de56"
Nov 02 01:57:27.104930 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:27.104876    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8fdaa36d00bd020581ca904b79d71c0a3f56213caf171f36143c802c286de56"} err="failed to get container status \"c8fdaa36d00bd020581ca904b79d71c0a3f56213caf171f36143c802c286de56\": rpc error: code = NotFound desc = could not find container \"c8fdaa36d00bd020581ca904b79d71c0a3f56213caf171f36143c802c286de56\": container with ID starting with c8fdaa36d00bd020581ca904b79d71c0a3f56213caf171f36143c802c286de56 not found: ID does not exist"
Nov 02 01:57:27.264532 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:27.264501    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q"]
Nov 02 01:57:27.272490 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:27.272465    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b9a2fcb6-7f9c7fbc5-h7h6q"]
Nov 02 01:57:28.695460 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:28.695430    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01885305-d407-4274-8c77-a393551d40cd" path="/var/lib/kubelet/pods/01885305-d407-4274-8c77-a393551d40cd/volumes"
Nov 02 01:57:31.168983 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.168947    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"]
Nov 02 01:57:31.169663 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.169648    5875 cpu_manager.go:414] "containerMap: removing container" podUID="4b31c334-b688-4f3a-9d08-d9b2e6ef3d33" containerName="revoke-keys"
Nov 02 01:57:31.169707 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.169665    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b31c334-b688-4f3a-9d08-d9b2e6ef3d33" containerName="revoke-keys"
Nov 02 01:57:31.169707 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.169693    5875 cpu_manager.go:414] "containerMap: removing container" podUID="01885305-d407-4274-8c77-a393551d40cd" containerName="payload-processing"
Nov 02 01:57:31.169707 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.169699    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01885305-d407-4274-8c77-a393551d40cd" containerName="payload-processing"
Nov 02 01:57:31.169827 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.169818    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="01885305-d407-4274-8c77-a393551d40cd" containerName="payload-processing"
Nov 02 01:57:31.169864 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.169836    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b31c334-b688-4f3a-9d08-d9b2e6ef3d33" containerName="revoke-keys"
Nov 02 01:57:31.175013 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.174994    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.218406 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.218381    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"]
Nov 02 01:57:31.222735 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.222718    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.289503 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.289471    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.289643 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.289526    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.289643 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.289628    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf75023b-badb-418a-a93f-48c3d7b0e562-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.289735 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.289658    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004e401a-4d50-46ec-b1bd-d9896f9b20d1-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.289735 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.289689    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mpb8\" (UniqueName: \"kubernetes.io/projected/004e401a-4d50-46ec-b1bd-d9896f9b20d1-kube-api-access-2mpb8\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.289848 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.289808    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.289961 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.289945    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004e401a-4d50-46ec-b1bd-d9896f9b20d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.290022 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.289996    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004e401a-4d50-46ec-b1bd-d9896f9b20d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.290079 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.290059    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004e401a-4d50-46ec-b1bd-d9896f9b20d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.290151 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.290134    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn9hz\" (UniqueName: \"kubernetes.io/projected/cf75023b-badb-418a-a93f-48c3d7b0e562-kube-api-access-zn9hz\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.290209 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.290168    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.290288 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.290214    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004e401a-4d50-46ec-b1bd-d9896f9b20d1-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.290343 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.290295    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.290407 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.290378    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.290460 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.290425    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.290460 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.290444    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004e401a-4d50-46ec-b1bd-d9896f9b20d1-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.290553 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.290539    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004e401a-4d50-46ec-b1bd-d9896f9b20d1-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.290665 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.290569    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004e401a-4d50-46ec-b1bd-d9896f9b20d1-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.329515 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.329482    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"]
Nov 02 01:57:31.378689 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.378659    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"]
Nov 02 01:57:31.392536 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.392509    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004e401a-4d50-46ec-b1bd-d9896f9b20d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.392727 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.392577    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004e401a-4d50-46ec-b1bd-d9896f9b20d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.392727 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.392609    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004e401a-4d50-46ec-b1bd-d9896f9b20d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.392727 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.392656    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn9hz\" (UniqueName: \"kubernetes.io/projected/cf75023b-badb-418a-a93f-48c3d7b0e562-kube-api-access-zn9hz\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.392727 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.392682    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.392727 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.392712    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004e401a-4d50-46ec-b1bd-d9896f9b20d1-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.392991 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.392760    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.392991 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.392836    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.392991 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.392876    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.392991 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.392906    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004e401a-4d50-46ec-b1bd-d9896f9b20d1-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.392991 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.392970    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004e401a-4d50-46ec-b1bd-d9896f9b20d1-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.393284 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.393020    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004e401a-4d50-46ec-b1bd-d9896f9b20d1-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.393284 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.393110    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.393284 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.393153    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.393284 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.393209    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf75023b-badb-418a-a93f-48c3d7b0e562-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.393284 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.393258    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004e401a-4d50-46ec-b1bd-d9896f9b20d1-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.393535 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.393297    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mpb8\" (UniqueName: \"kubernetes.io/projected/004e401a-4d50-46ec-b1bd-d9896f9b20d1-kube-api-access-2mpb8\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.393535 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.393333    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.393535 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.393509    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004e401a-4d50-46ec-b1bd-d9896f9b20d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.393699 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.393562    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.395116 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.393781    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.395116 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.394073    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.395116 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.395047    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004e401a-4d50-46ec-b1bd-d9896f9b20d1-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.395780 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.395758    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004e401a-4d50-46ec-b1bd-d9896f9b20d1-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.396475 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.395829    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.397263 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.397239    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004e401a-4d50-46ec-b1bd-d9896f9b20d1-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.397452 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.397434    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf75023b-badb-418a-a93f-48c3d7b0e562-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.397561 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.397479    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004e401a-4d50-46ec-b1bd-d9896f9b20d1-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.398027 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.398004    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.399984 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.398772    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004e401a-4d50-46ec-b1bd-d9896f9b20d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.399984 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.398964    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.400688 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.400665    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004e401a-4d50-46ec-b1bd-d9896f9b20d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.454859 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.454836    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004e401a-4d50-46ec-b1bd-d9896f9b20d1-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.455354 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.455332    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn9hz\" (UniqueName: \"kubernetes.io/projected/cf75023b-badb-418a-a93f-48c3d7b0e562-kube-api-access-zn9hz\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.456542 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.456515    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ts98c\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.457328 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.457307    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mpb8\" (UniqueName: \"kubernetes.io/projected/004e401a-4d50-46ec-b1bd-d9896f9b20d1-kube-api-access-2mpb8\") pod \"maas-default-gateway-openshift-default-778b7b5c47-mfmmj\" (UID: \"004e401a-4d50-46ec-b1bd-d9896f9b20d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.489516 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.489491    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:31.536842 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.536814    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:31.928427 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.928355    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"]
Nov 02 01:57:31.962649 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:57:31.962617    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf75023b_badb_418a_a93f_48c3d7b0e562.slice/crio-5f5742337d6d3eb11285aa25772da1a7a9a2418adfcba0882c6187c52201b398 WatchSource:0}: Error finding container 5f5742337d6d3eb11285aa25772da1a7a9a2418adfcba0882c6187c52201b398: Status 404 returned error can't find the container with id 5f5742337d6d3eb11285aa25772da1a7a9a2418adfcba0882c6187c52201b398
Nov 02 01:57:31.965289 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.965249    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:57:31.965376 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.965325    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:57:31.965376 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.965354    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:57:31.980861 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:57:31.980834    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004e401a_4d50_46ec_b1bd_d9896f9b20d1.slice/crio-dc36692b80f340506668018a4ac1401c11553d81d92ff630da83a99ba96c9c0a WatchSource:0}: Error finding container dc36692b80f340506668018a4ac1401c11553d81d92ff630da83a99ba96c9c0a: Status 404 returned error can't find the container with id dc36692b80f340506668018a4ac1401c11553d81d92ff630da83a99ba96c9c0a
Nov 02 01:57:31.981493 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.981472    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"]
Nov 02 01:57:31.983560 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.983525    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:57:31.983643 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.983590    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:57:31.983643 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:31.983620    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:57:32.140289 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:32.139724    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c" event={"ID":"cf75023b-badb-418a-a93f-48c3d7b0e562","Type":"ContainerStarted","Data":"c32111e903554f1735e50447faead4ea5b83daedda360f57b77588d16d21456c"}
Nov 02 01:57:32.140289 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:32.139781    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c" event={"ID":"cf75023b-badb-418a-a93f-48c3d7b0e562","Type":"ContainerStarted","Data":"5f5742337d6d3eb11285aa25772da1a7a9a2418adfcba0882c6187c52201b398"}
Nov 02 01:57:32.142481 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:32.142456    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj" event={"ID":"004e401a-4d50-46ec-b1bd-d9896f9b20d1","Type":"ContainerStarted","Data":"9258714d5924e40044fe1f586a94c1dbc42fbabba3e861d13f77ccd323406146"}
Nov 02 01:57:32.142566 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:32.142499    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj" event={"ID":"004e401a-4d50-46ec-b1bd-d9896f9b20d1","Type":"ContainerStarted","Data":"dc36692b80f340506668018a4ac1401c11553d81d92ff630da83a99ba96c9c0a"}
Nov 02 01:57:32.266252 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:32.265979    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c" podStartSLOduration=1.265959062 podStartE2EDuration="1.265959062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:57:32.258583006 +0000 UTC m=+1499.699619498" watchObservedRunningTime="2026-11-02 01:57:32.265959062 +0000 UTC m=+1499.706995563"
Nov 02 01:57:32.382423 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:32.382360    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj" podStartSLOduration=1.3823422060000001 podStartE2EDuration="1.382342206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:57:32.378422924 +0000 UTC m=+1499.819459422" watchObservedRunningTime="2026-11-02 01:57:32.382342206 +0000 UTC m=+1499.823378716"
Nov 02 01:57:32.489769 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:32.489734    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:32.537272 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:32.537153    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:33.491299 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:33.491263    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ts98c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 02 01:57:33.491693 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:33.491322    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c" podUID="cf75023b-badb-418a-a93f-48c3d7b0e562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 02 01:57:33.549247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:33.548409    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-mfmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 01:57:33.549247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:33.548467    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj" podUID="004e401a-4d50-46ec-b1bd-d9896f9b20d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 01:57:34.500277 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:34.498196    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ts98c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 02 01:57:34.500277 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:34.498281    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c" podUID="cf75023b-badb-418a-a93f-48c3d7b0e562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 02 01:57:34.726385 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:34.720085    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-mfmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 01:57:34.726385 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:34.720151    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj" podUID="004e401a-4d50-46ec-b1bd-d9896f9b20d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 01:57:35.531248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:35.522853    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ts98c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 02 01:57:35.531248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:35.522920    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c" podUID="cf75023b-badb-418a-a93f-48c3d7b0e562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 02 01:57:35.549441 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:35.548720    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-mfmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 01:57:35.549441 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:35.548798    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj" podUID="004e401a-4d50-46ec-b1bd-d9896f9b20d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 01:57:36.494333 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:36.494297    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ts98c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 02 01:57:36.495162 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:36.495126    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c" podUID="cf75023b-badb-418a-a93f-48c3d7b0e562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 02 01:57:36.815296 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:36.810842    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-mfmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 01:57:36.815296 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:36.810919    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj" podUID="004e401a-4d50-46ec-b1bd-d9896f9b20d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 01:57:37.043254 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:37.040896    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15487453-s829c"]
Nov 02 01:57:37.043254 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:37.040958    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15487453-s829c"]
Nov 02 01:57:37.043254 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:37.040985    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"]
Nov 02 01:57:37.043254 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:37.041208    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf" podUID="222ac2e9-73cf-4ad3-8704-807adb22fb1a" containerName="istio-proxy" containerID="cri-o://749ab0743456c2cc5bb61862d893387d8350b7373b7ef8f572a91b5e10d24310" gracePeriod=30
Nov 02 01:57:37.490449 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:37.490422    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ts98c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 02 01:57:37.490554 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:37.490479    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c" podUID="cf75023b-badb-418a-a93f-48c3d7b0e562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 02 01:57:37.554241 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:37.554190    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-mfmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 01:57:37.554531 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:37.554485    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj" podUID="004e401a-4d50-46ec-b1bd-d9896f9b20d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 01:57:37.701249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:37.697285    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"]
Nov 02 01:57:37.718778 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:37.718706    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:37.781920 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:37.773760    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4f2f926d-openshift-default-dockercfg-52phl\""
Nov 02 01:57:37.828660 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:37.828617    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-workload-certs\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:37.914245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:37.904329    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-workload-socket\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:37.914245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:37.904417    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-istio-envoy\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:37.914245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:37.904760    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98296768-250b-4160-8678-b92e2fb3a151-istio-podinfo\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:37.914245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:37.904830    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98296768-250b-4160-8678-b92e2fb3a151-istio-token\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:37.914245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:37.904866    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-istio-data\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:37.914245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:37.904954    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98296768-250b-4160-8678-b92e2fb3a151-istiod-ca-cert\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:37.914245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:37.905027    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-credential-socket\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:37.914245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:37.905166    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9c2x\" (UniqueName: \"kubernetes.io/projected/98296768-250b-4160-8678-b92e2fb3a151-kube-api-access-l9c2x\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:37.914245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:37.860363    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"]
Nov 02 01:57:38.078289 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:38.053584    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9c2x\" (UniqueName: \"kubernetes.io/projected/98296768-250b-4160-8678-b92e2fb3a151-kube-api-access-l9c2x\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:38.164906 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:38.145359    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-workload-certs\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:38.164906 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:38.145440    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-workload-socket\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:38.164906 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:38.145526    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-istio-envoy\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:38.164906 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:38.145786    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98296768-250b-4160-8678-b92e2fb3a151-istio-podinfo\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:38.164906 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:38.145859    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98296768-250b-4160-8678-b92e2fb3a151-istio-token\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:38.164906 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:38.145900    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-istio-data\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:38.164906 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:38.145974    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98296768-250b-4160-8678-b92e2fb3a151-istiod-ca-cert\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:38.164906 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:38.146038    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-credential-socket\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:38.164906 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:38.146678    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-credential-socket\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:38.164906 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:38.146956    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-workload-certs\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:38.164906 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:38.147255    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-workload-socket\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:38.164906 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:38.163591    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-istio-data\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:38.164906 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:38.164168    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98296768-250b-4160-8678-b92e2fb3a151-istiod-ca-cert\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:38.233113 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:38.233079    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-istio-envoy\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:38.264003 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:38.263903    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9c2x\" (UniqueName: \"kubernetes.io/projected/98296768-250b-4160-8678-b92e2fb3a151-kube-api-access-l9c2x\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:38.280452 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:38.280421    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98296768-250b-4160-8678-b92e2fb3a151-istio-token\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:38.281291 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:38.281214    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98296768-250b-4160-8678-b92e2fb3a151-istio-podinfo\") pod \"e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") " pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:38.491402 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:38.491181    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ts98c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 02 01:57:38.491402 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:38.491255    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c" podUID="cf75023b-badb-418a-a93f-48c3d7b0e562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 02 01:57:38.528791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:38.527555    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:38.547783 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:38.547752    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-mfmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 01:57:38.548012 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:38.547983    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj" podUID="004e401a-4d50-46ec-b1bd-d9896f9b20d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 01:57:38.869352 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:38.868649    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5358ac77-8542-4ae0-99e9-4d482e3fca7e" path="/var/lib/kubelet/pods/5358ac77-8542-4ae0-99e9-4d482e3fca7e/volumes"
Nov 02 01:57:39.262529 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:57:39.258196    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98296768_250b_4160_8678_b92e2fb3a151.slice/crio-0927580bdf5b304f3e25a6506fc40b57cd8965e4cfbc7bd7ba84f4a9caa337ce WatchSource:0}: Error finding container 0927580bdf5b304f3e25a6506fc40b57cd8965e4cfbc7bd7ba84f4a9caa337ce: Status 404 returned error can't find the container with id 0927580bdf5b304f3e25a6506fc40b57cd8965e4cfbc7bd7ba84f4a9caa337ce
Nov 02 01:57:39.262529 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:39.259929    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"]
Nov 02 01:57:39.264514 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:39.264467    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:57:39.264689 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:39.264546    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:57:39.264689 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:39.264587    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:57:39.470309 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:39.470274    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz" event={"ID":"98296768-250b-4160-8678-b92e2fb3a151","Type":"ContainerStarted","Data":"22a9264bd402ba32e032bfc65f004c848e702ebac53f1a5c3f974db876bf995f"}
Nov 02 01:57:39.470466 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:39.470337    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz" event={"ID":"98296768-250b-4160-8678-b92e2fb3a151","Type":"ContainerStarted","Data":"0927580bdf5b304f3e25a6506fc40b57cd8965e4cfbc7bd7ba84f4a9caa337ce"}
Nov 02 01:57:39.489791 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:39.489759    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ts98c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 02 01:57:39.489970 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:39.489812    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c" podUID="cf75023b-badb-418a-a93f-48c3d7b0e562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 02 01:57:39.541304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:39.540620    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-mfmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 01:57:39.541304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:39.540693    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj" podUID="004e401a-4d50-46ec-b1bd-d9896f9b20d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 01:57:39.559245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:39.553420    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:39.594139 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:39.594089    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz" podStartSLOduration=2.594074331 podStartE2EDuration="2.594074331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:57:39.590185723 +0000 UTC m=+1507.031222221" watchObservedRunningTime="2026-11-02 01:57:39.594074331 +0000 UTC m=+1507.035110828"
Nov 02 01:57:40.490053 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:40.490020    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ts98c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 02 01:57:40.490483 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:40.490066    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c" podUID="cf75023b-badb-418a-a93f-48c3d7b0e562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 02 01:57:40.537872 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:40.537838    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-mfmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 01:57:40.538024 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:40.537899    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj" podUID="004e401a-4d50-46ec-b1bd-d9896f9b20d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 01:57:40.553636 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:40.553613    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:41.323833 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:57:41.323775    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:57:41.490483 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:41.490447    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ts98c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 02 01:57:41.491019 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:41.490508    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c" podUID="cf75023b-badb-418a-a93f-48c3d7b0e562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 02 01:57:41.491019 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:41.490452    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:41.491528 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:41.491508    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:57:41.550249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:41.540619    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-mfmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 02 01:57:41.550249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:41.540706    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj" podUID="004e401a-4d50-46ec-b1bd-d9896f9b20d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 02 01:57:42.507056 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:42.507014    5875 generic.go:356] "Generic (PLEG): container finished" podID="222ac2e9-73cf-4ad3-8704-807adb22fb1a" containerID="749ab0743456c2cc5bb61862d893387d8350b7373b7ef8f572a91b5e10d24310" exitCode=0
Nov 02 01:57:42.507698 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:42.507080    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf" event={"ID":"222ac2e9-73cf-4ad3-8704-807adb22fb1a","Type":"ContainerDied","Data":"749ab0743456c2cc5bb61862d893387d8350b7373b7ef8f572a91b5e10d24310"}
Nov 02 01:57:42.507698 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:42.507191    5875 scope.go:122] "RemoveContainer" containerID="c2373a48e2a426084eb3835bc8e26371b5176832c5f307d58be2d610b8b8df31"
Nov 02 01:57:43.211548 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.211481    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:57:43.339996 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.339912    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istiod-ca-cert\") pod \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") "
Nov 02 01:57:43.340345 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.340321    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-workload-certs\") pod \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") "
Nov 02 01:57:43.340507 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.340490    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-token\" (UniqueName: \"kubernetes.io/projected/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-token\") pod \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") "
Nov 02 01:57:43.340708 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.340541    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-envoy\") pod \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") "
Nov 02 01:57:43.340708 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.340334    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istiod-ca-cert" pod "222ac2e9-73cf-4ad3-8704-807adb22fb1a" (UID: "222ac2e9-73cf-4ad3-8704-807adb22fb1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:57:43.340708 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.340580    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-workload-certs" pod "222ac2e9-73cf-4ad3-8704-807adb22fb1a" (UID: "222ac2e9-73cf-4ad3-8704-807adb22fb1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:43.340914 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.340724    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/222ac2e9-73cf-4ad3-8704-807adb22fb1a-kube-api-access-6mh8z\" (UniqueName: \"kubernetes.io/projected/222ac2e9-73cf-4ad3-8704-807adb22fb1a-kube-api-access-6mh8z\") pod \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") "
Nov 02 01:57:43.340914 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.340821    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-credential-socket\") pod \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") "
Nov 02 01:57:43.340914 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.340879    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-data\") pod \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") "
Nov 02 01:57:43.341061 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.340966    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-workload-socket\") pod \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") "
Nov 02 01:57:43.341061 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.341021    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-podinfo\") pod \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\" (UID: \"222ac2e9-73cf-4ad3-8704-807adb22fb1a\") "
Nov 02 01:57:43.341464 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.341416    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-credential-socket" pod "222ac2e9-73cf-4ad3-8704-807adb22fb1a" (UID: "222ac2e9-73cf-4ad3-8704-807adb22fb1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:43.341571 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.341529    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-data" pod "222ac2e9-73cf-4ad3-8704-807adb22fb1a" (UID: "222ac2e9-73cf-4ad3-8704-807adb22fb1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:43.342082 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.341973    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:43.342082 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.341999    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:43.342082 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.342014    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:43.342082 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.342029    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:43.342082 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.342089    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-workload-socket" pod "222ac2e9-73cf-4ad3-8704-807adb22fb1a" (UID: "222ac2e9-73cf-4ad3-8704-807adb22fb1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:43.345059 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.345031    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-token" pod "222ac2e9-73cf-4ad3-8704-807adb22fb1a" (UID: "222ac2e9-73cf-4ad3-8704-807adb22fb1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:43.345059 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.345042    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-envoy" pod "222ac2e9-73cf-4ad3-8704-807adb22fb1a" (UID: "222ac2e9-73cf-4ad3-8704-807adb22fb1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:43.345193 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.345153    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/222ac2e9-73cf-4ad3-8704-807adb22fb1a-kube-api-access-6mh8z" pod "222ac2e9-73cf-4ad3-8704-807adb22fb1a" (UID: "222ac2e9-73cf-4ad3-8704-807adb22fb1a"). InnerVolumeSpecName "kube-api-access-6mh8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:43.345467 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.345438    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-podinfo" pod "222ac2e9-73cf-4ad3-8704-807adb22fb1a" (UID: "222ac2e9-73cf-4ad3-8704-807adb22fb1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:57:43.444276 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.444193    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:43.444598 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.444516    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:43.444598 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.444545    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mh8z\" (UniqueName: \"kubernetes.io/projected/222ac2e9-73cf-4ad3-8704-807adb22fb1a-kube-api-access-6mh8z\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:43.444598 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.444562    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/222ac2e9-73cf-4ad3-8704-807adb22fb1a-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:43.444598 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.444578    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/222ac2e9-73cf-4ad3-8704-807adb22fb1a-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:43.492291 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.490602    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ts98c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:57:43.492291 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.490685    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c" podUID="cf75023b-badb-418a-a93f-48c3d7b0e562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:57:43.521598 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.521564    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"
Nov 02 01:57:43.521598 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.521581    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf" event={"ID":"222ac2e9-73cf-4ad3-8704-807adb22fb1a","Type":"ContainerDied","Data":"f0e4bba7d524db4f471e21f491de269b8c051998b1b16192bc06f79d704de8bd"}
Nov 02 01:57:43.522095 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.521655    5875 scope.go:122] "RemoveContainer" containerID="749ab0743456c2cc5bb61862d893387d8350b7373b7ef8f572a91b5e10d24310"
Nov 02 01:57:43.538583 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.538556    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-mfmmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:57:43.538757 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.538612    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj" podUID="004e401a-4d50-46ec-b1bd-d9896f9b20d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:57:43.545190 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.544321    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:43.550871 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.550838    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:43.552931 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.552907    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 01:57:43.574812 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.574777    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:43.575339 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.575306    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:43.576143 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.576124    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj"
Nov 02 01:57:43.690354 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.690148    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"]
Nov 02 01:57:43.697251 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:43.697190    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-15487453-openshift-default-586d57b8bc-2z6bf"]
Nov 02 01:57:44.696379 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:44.696351    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="222ac2e9-73cf-4ad3-8704-807adb22fb1a" path="/var/lib/kubelet/pods/222ac2e9-73cf-4ad3-8704-807adb22fb1a/volumes"
Nov 02 01:57:45.160255 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:45.147795    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp"]
Nov 02 01:57:45.193247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:45.189571    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-495b48c2-f8sxp"]
Nov 02 01:57:45.682095 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:45.672959    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"]
Nov 02 01:57:45.682095 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:45.673392    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5" podUID="a47b2793-7e4c-4cae-8d55-6bc82c1e4006" containerName="istio-proxy" containerID="cri-o://5df611d78c9c4dd82293ccc0bd56e09e53a8bcb04b8b86b4c157016ada6d36c5" gracePeriod=30
Nov 02 01:57:46.699097 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:46.699058    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3cf00e4-8887-4f12-b0fa-5566ad05e65c" path="/var/lib/kubelet/pods/c3cf00e4-8887-4f12-b0fa-5566ad05e65c/volumes"
Nov 02 01:57:47.282269 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.281035    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"]
Nov 02 01:57:47.282472 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.282342    5875 cpu_manager.go:414] "containerMap: removing container" podUID="222ac2e9-73cf-4ad3-8704-807adb22fb1a" containerName="istio-proxy"
Nov 02 01:57:47.282472 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.282361    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="222ac2e9-73cf-4ad3-8704-807adb22fb1a" containerName="istio-proxy"
Nov 02 01:57:47.282472 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.282387    5875 cpu_manager.go:414] "containerMap: removing container" podUID="222ac2e9-73cf-4ad3-8704-807adb22fb1a" containerName="istio-proxy"
Nov 02 01:57:47.282472 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.282394    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="222ac2e9-73cf-4ad3-8704-807adb22fb1a" containerName="istio-proxy"
Nov 02 01:57:47.282688 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.282624    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="222ac2e9-73cf-4ad3-8704-807adb22fb1a" containerName="istio-proxy"
Nov 02 01:57:47.282688 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.282642    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="222ac2e9-73cf-4ad3-8704-807adb22fb1a" containerName="istio-proxy"
Nov 02 01:57:47.318926 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.318900    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.319388 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.319130    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"]
Nov 02 01:57:47.372610 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.372573    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-af807eae-openshift-default-dockercfg-j744x\""
Nov 02 01:57:47.469814 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.469776    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istiod-ca-cert\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.470140 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.470116    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-credential-socket\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.470312 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.470295    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2k7d\" (UniqueName: \"kubernetes.io/projected/ff6cb777-acc6-4c58-8302-26d6e2906cdf-kube-api-access-t2k7d\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.470514 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.470493    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-workload-certs\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.470677 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.470660    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-token\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.470820 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.470803    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-data\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.470997 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.470961    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-workload-socket\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.471174 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.471144    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-podinfo\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.471446 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.471424    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-envoy\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.576350 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.576261    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-workload-certs\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.577056 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.577034    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-token\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.577289 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.577270    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-data\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.577451 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.577437    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-workload-socket\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.577602 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.577584    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-podinfo\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.577817 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.577791    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-envoy\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.578040 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.578021    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istiod-ca-cert\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.578283 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.578264    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-credential-socket\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.580298 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.578430    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2k7d\" (UniqueName: \"kubernetes.io/projected/ff6cb777-acc6-4c58-8302-26d6e2906cdf-kube-api-access-t2k7d\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.580877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.576936    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-workload-certs\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.581817 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.581534    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-data\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.582370 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.582331    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-workload-socket\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.583696 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.583641    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-credential-socket\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.585356 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.585324    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istiod-ca-cert\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.606291 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.600842    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-podinfo\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.606558 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.606534    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-envoy\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.645796 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.645767    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2k7d\" (UniqueName: \"kubernetes.io/projected/ff6cb777-acc6-4c58-8302-26d6e2906cdf-kube-api-access-t2k7d\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.646674 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.646651    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-token\") pod \"e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") " pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:47.682237 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:47.682170    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:48.241984 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:57:48.241955    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6cb777_acc6_4c58_8302_26d6e2906cdf.slice/crio-3552f1e88dd16e273e586c8388d1fdea3c38a80c89e9c35c857288af4df5001b WatchSource:0}: Error finding container 3552f1e88dd16e273e586c8388d1fdea3c38a80c89e9c35c857288af4df5001b: Status 404 returned error can't find the container with id 3552f1e88dd16e273e586c8388d1fdea3c38a80c89e9c35c857288af4df5001b
Nov 02 01:57:48.242344 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:48.242303    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"]
Nov 02 01:57:48.244670 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:48.244637    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:57:48.244749 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:48.244705    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:57:48.244749 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:48.244735    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:57:48.646758 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:48.646670    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb" event={"ID":"ff6cb777-acc6-4c58-8302-26d6e2906cdf","Type":"ContainerStarted","Data":"1af18b7499f5362c05ea1b91c9d3b741f1bf47f262cd701f6a6b9e10e0c3c86d"}
Nov 02 01:57:48.646758 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:48.646728    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb" event={"ID":"ff6cb777-acc6-4c58-8302-26d6e2906cdf","Type":"ContainerStarted","Data":"3552f1e88dd16e273e586c8388d1fdea3c38a80c89e9c35c857288af4df5001b"}
Nov 02 01:57:48.683395 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:48.683367    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:48.778145 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:48.778079    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb" podStartSLOduration=1.778060066 podStartE2EDuration="1.778060066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:57:48.768266332 +0000 UTC m=+1516.209302842" watchObservedRunningTime="2026-11-02 01:57:48.778060066 +0000 UTC m=+1516.219096564"
Nov 02 01:57:49.688660 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:49.688631    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:50.664649 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:50.664623    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:50.665554 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:50.665533    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:57:51.102458 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.102425    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb"]
Nov 02 01:57:51.110102 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.110073    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b9a2fcb6-xkrxb"]
Nov 02 01:57:51.205328 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.205290    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"]
Nov 02 01:57:51.205593 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.205565    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf" podUID="013af201-2be2-4d4a-b07f-edbfd3ec1ac5" containerName="istio-proxy" containerID="cri-o://509e5d228c9c4555f90288f5d15fd0cf1abedbdd49b5ca945ad31a334deeffa4" gracePeriod=30
Nov 02 01:57:51.215947 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.215926    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:57:51.318245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.314852    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-kube-api-access-fkr4m\" (UniqueName: \"kubernetes.io/projected/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-kube-api-access-fkr4m\") pod \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") "
Nov 02 01:57:51.318245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.314924    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-workload-certs\") pod \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") "
Nov 02 01:57:51.318245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.315139    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-data\") pod \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") "
Nov 02 01:57:51.318245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.315246    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-token\" (UniqueName: \"kubernetes.io/projected/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-token\") pod \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") "
Nov 02 01:57:51.318245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.315277    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-envoy\") pod \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") "
Nov 02 01:57:51.318245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.315368    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istiod-ca-cert\") pod \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") "
Nov 02 01:57:51.318245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.315431    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-workload-socket\") pod \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") "
Nov 02 01:57:51.318245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.315552    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-podinfo\") pod \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") "
Nov 02 01:57:51.318245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.315625    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-credential-socket\") pod \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\" (UID: \"a47b2793-7e4c-4cae-8d55-6bc82c1e4006\") "
Nov 02 01:57:51.368380 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.350821    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-credential-socket" pod "a47b2793-7e4c-4cae-8d55-6bc82c1e4006" (UID: "a47b2793-7e4c-4cae-8d55-6bc82c1e4006"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:51.368380 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.356312    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-workload-certs" pod "a47b2793-7e4c-4cae-8d55-6bc82c1e4006" (UID: "a47b2793-7e4c-4cae-8d55-6bc82c1e4006"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:51.368380 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.359035    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-workload-socket" pod "a47b2793-7e4c-4cae-8d55-6bc82c1e4006" (UID: "a47b2793-7e4c-4cae-8d55-6bc82c1e4006"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:51.368380 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.359211    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istiod-ca-cert" pod "a47b2793-7e4c-4cae-8d55-6bc82c1e4006" (UID: "a47b2793-7e4c-4cae-8d55-6bc82c1e4006"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:57:51.368905 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.368806    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-data" pod "a47b2793-7e4c-4cae-8d55-6bc82c1e4006" (UID: "a47b2793-7e4c-4cae-8d55-6bc82c1e4006"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:51.413497 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.409421    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-token" pod "a47b2793-7e4c-4cae-8d55-6bc82c1e4006" (UID: "a47b2793-7e4c-4cae-8d55-6bc82c1e4006"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:51.413497 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.411387    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-kube-api-access-fkr4m" pod "a47b2793-7e4c-4cae-8d55-6bc82c1e4006" (UID: "a47b2793-7e4c-4cae-8d55-6bc82c1e4006"). InnerVolumeSpecName "kube-api-access-fkr4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:51.413497 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.412581    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-envoy" pod "a47b2793-7e4c-4cae-8d55-6bc82c1e4006" (UID: "a47b2793-7e4c-4cae-8d55-6bc82c1e4006"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:51.414442 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.414396    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-podinfo" pod "a47b2793-7e4c-4cae-8d55-6bc82c1e4006" (UID: "a47b2793-7e4c-4cae-8d55-6bc82c1e4006"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:57:51.426307 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.419722    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:51.435270 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.435213    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:51.435508 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.435491    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:51.435682 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.435588    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:51.435682 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.435606    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:51.435682 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.435621    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:51.435682 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.435633    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:51.435682 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.435649    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkr4m\" (UniqueName: \"kubernetes.io/projected/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-kube-api-access-fkr4m\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:51.435682 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.435663    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a47b2793-7e4c-4cae-8d55-6bc82c1e4006-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:51.689577 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.689503    5875 generic.go:356] "Generic (PLEG): container finished" podID="a47b2793-7e4c-4cae-8d55-6bc82c1e4006" containerID="5df611d78c9c4dd82293ccc0bd56e09e53a8bcb04b8b86b4c157016ada6d36c5" exitCode=0
Nov 02 01:57:51.689577 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.689557    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"
Nov 02 01:57:51.689786 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.689583    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5" event={"ID":"a47b2793-7e4c-4cae-8d55-6bc82c1e4006","Type":"ContainerDied","Data":"5df611d78c9c4dd82293ccc0bd56e09e53a8bcb04b8b86b4c157016ada6d36c5"}
Nov 02 01:57:51.689786 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.689633    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5" event={"ID":"a47b2793-7e4c-4cae-8d55-6bc82c1e4006","Type":"ContainerDied","Data":"91c9cd61126fea33741b7bac77e75727f20d314e7c65f4753f9b5609fe7085f8"}
Nov 02 01:57:51.689786 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.689660    5875 scope.go:122] "RemoveContainer" containerID="5df611d78c9c4dd82293ccc0bd56e09e53a8bcb04b8b86b4c157016ada6d36c5"
Nov 02 01:57:51.709499 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.709480    5875 scope.go:122] "RemoveContainer" containerID="352ba78804bb01a34735761f5821a644d72cfe16add40b61bbb42bcd1c5f0089"
Nov 02 01:57:51.720584 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.720558    5875 scope.go:122] "RemoveContainer" containerID="5df611d78c9c4dd82293ccc0bd56e09e53a8bcb04b8b86b4c157016ada6d36c5"
Nov 02 01:57:51.720899 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:57:51.720880    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df611d78c9c4dd82293ccc0bd56e09e53a8bcb04b8b86b4c157016ada6d36c5\": container with ID starting with 5df611d78c9c4dd82293ccc0bd56e09e53a8bcb04b8b86b4c157016ada6d36c5 not found: ID does not exist" containerID="5df611d78c9c4dd82293ccc0bd56e09e53a8bcb04b8b86b4c157016ada6d36c5"
Nov 02 01:57:51.720962 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.720912    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df611d78c9c4dd82293ccc0bd56e09e53a8bcb04b8b86b4c157016ada6d36c5"} err="failed to get container status \"5df611d78c9c4dd82293ccc0bd56e09e53a8bcb04b8b86b4c157016ada6d36c5\": rpc error: code = NotFound desc = could not find container \"5df611d78c9c4dd82293ccc0bd56e09e53a8bcb04b8b86b4c157016ada6d36c5\": container with ID starting with 5df611d78c9c4dd82293ccc0bd56e09e53a8bcb04b8b86b4c157016ada6d36c5 not found: ID does not exist"
Nov 02 01:57:51.720962 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.720930    5875 scope.go:122] "RemoveContainer" containerID="352ba78804bb01a34735761f5821a644d72cfe16add40b61bbb42bcd1c5f0089"
Nov 02 01:57:51.721174 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:57:51.721160    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"352ba78804bb01a34735761f5821a644d72cfe16add40b61bbb42bcd1c5f0089\": container with ID starting with 352ba78804bb01a34735761f5821a644d72cfe16add40b61bbb42bcd1c5f0089 not found: ID does not exist" containerID="352ba78804bb01a34735761f5821a644d72cfe16add40b61bbb42bcd1c5f0089"
Nov 02 01:57:51.721214 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.721178    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"352ba78804bb01a34735761f5821a644d72cfe16add40b61bbb42bcd1c5f0089"} err="failed to get container status \"352ba78804bb01a34735761f5821a644d72cfe16add40b61bbb42bcd1c5f0089\": rpc error: code = NotFound desc = could not find container \"352ba78804bb01a34735761f5821a644d72cfe16add40b61bbb42bcd1c5f0089\": container with ID starting with 352ba78804bb01a34735761f5821a644d72cfe16add40b61bbb42bcd1c5f0089 not found: ID does not exist"
Nov 02 01:57:51.860249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.860203    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"]
Nov 02 01:57:51.867936 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:51.867906    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-495b48c2-openshift-default-6b48866c75-sr7q5"]
Nov 02 01:57:52.698172 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:52.698127    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b31c334-b688-4f3a-9d08-d9b2e6ef3d33" path="/var/lib/kubelet/pods/4b31c334-b688-4f3a-9d08-d9b2e6ef3d33/volumes"
Nov 02 01:57:52.698858 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:52.698835    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a47b2793-7e4c-4cae-8d55-6bc82c1e4006" path="/var/lib/kubelet/pods/a47b2793-7e4c-4cae-8d55-6bc82c1e4006/volumes"
Nov 02 01:57:54.131285 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:54.131249    5875 patch_prober.go:29] interesting pod/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:57:54.131662 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:54.131318    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf" podUID="013af201-2be2-4d4a-b07f-edbfd3ec1ac5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:57:56.517987 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.517962    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:57:56.574248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.572650    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-token\" (UniqueName: \"kubernetes.io/projected/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-token\") pod \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") "
Nov 02 01:57:56.574248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.573184    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istiod-ca-cert\") pod \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") "
Nov 02 01:57:56.574248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.573836    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istiod-ca-cert" pod "013af201-2be2-4d4a-b07f-edbfd3ec1ac5" (UID: "013af201-2be2-4d4a-b07f-edbfd3ec1ac5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:57:56.589919 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.589859    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-token" pod "013af201-2be2-4d4a-b07f-edbfd3ec1ac5" (UID: "013af201-2be2-4d4a-b07f-edbfd3ec1ac5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:56.607597 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.590913    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-kube-api-access-gxrk7\" (UniqueName: \"kubernetes.io/projected/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-kube-api-access-gxrk7\") pod \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") "
Nov 02 01:57:56.607597 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.591052    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-workload-certs\") pod \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") "
Nov 02 01:57:56.607597 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.591187    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-podinfo\") pod \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") "
Nov 02 01:57:56.607597 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.591358    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-credential-socket\") pod \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") "
Nov 02 01:57:56.607597 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.591413    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-workload-socket\") pod \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") "
Nov 02 01:57:56.607597 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.591583    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-data\") pod \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") "
Nov 02 01:57:56.607597 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.591667    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-envoy\") pod \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\" (UID: \"013af201-2be2-4d4a-b07f-edbfd3ec1ac5\") "
Nov 02 01:57:56.607597 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.598612    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-credential-socket" pod "013af201-2be2-4d4a-b07f-edbfd3ec1ac5" (UID: "013af201-2be2-4d4a-b07f-edbfd3ec1ac5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:56.607597 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.599715    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:56.607597 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.599745    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:56.607597 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.599761    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:56.607597 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.602311    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-data" pod "013af201-2be2-4d4a-b07f-edbfd3ec1ac5" (UID: "013af201-2be2-4d4a-b07f-edbfd3ec1ac5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:56.607597 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.602403    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-workload-certs" pod "013af201-2be2-4d4a-b07f-edbfd3ec1ac5" (UID: "013af201-2be2-4d4a-b07f-edbfd3ec1ac5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:56.607597 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.602454    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-workload-socket" pod "013af201-2be2-4d4a-b07f-edbfd3ec1ac5" (UID: "013af201-2be2-4d4a-b07f-edbfd3ec1ac5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:56.622085 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.617305    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-kube-api-access-gxrk7" pod "013af201-2be2-4d4a-b07f-edbfd3ec1ac5" (UID: "013af201-2be2-4d4a-b07f-edbfd3ec1ac5"). InnerVolumeSpecName "kube-api-access-gxrk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:56.636245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.633747    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-envoy" pod "013af201-2be2-4d4a-b07f-edbfd3ec1ac5" (UID: "013af201-2be2-4d4a-b07f-edbfd3ec1ac5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:56.652142 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.652093    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-podinfo" pod "013af201-2be2-4d4a-b07f-edbfd3ec1ac5" (UID: "013af201-2be2-4d4a-b07f-edbfd3ec1ac5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:57:56.701872 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.701821    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxrk7\" (UniqueName: \"kubernetes.io/projected/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-kube-api-access-gxrk7\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:56.701872 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.701858    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:56.701872 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.701873    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:56.702374 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.701886    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:56.702374 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.702376    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:56.702571 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.702391    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013af201-2be2-4d4a-b07f-edbfd3ec1ac5-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:56.977704 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.977670    5875 generic.go:356] "Generic (PLEG): container finished" podID="013af201-2be2-4d4a-b07f-edbfd3ec1ac5" containerID="509e5d228c9c4555f90288f5d15fd0cf1abedbdd49b5ca945ad31a334deeffa4" exitCode=0
Nov 02 01:57:56.977869 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.977723    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"
Nov 02 01:57:56.977869 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.977748    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf" event={"ID":"013af201-2be2-4d4a-b07f-edbfd3ec1ac5","Type":"ContainerDied","Data":"509e5d228c9c4555f90288f5d15fd0cf1abedbdd49b5ca945ad31a334deeffa4"}
Nov 02 01:57:56.977869 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.977804    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf" event={"ID":"013af201-2be2-4d4a-b07f-edbfd3ec1ac5","Type":"ContainerDied","Data":"b01011e27dab90bf0aa357718ce40217b772ca1d32bb232d32ee2d1a1dffe666"}
Nov 02 01:57:56.977869 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.977833    5875 scope.go:122] "RemoveContainer" containerID="509e5d228c9c4555f90288f5d15fd0cf1abedbdd49b5ca945ad31a334deeffa4"
Nov 02 01:57:56.988846 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.988829    5875 scope.go:122] "RemoveContainer" containerID="f8ab5f234644ab5b25f9760f9c8c0c228d4d09595ea95ff6cd8ef737e7e31fa5"
Nov 02 01:57:56.998683 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.998665    5875 scope.go:122] "RemoveContainer" containerID="509e5d228c9c4555f90288f5d15fd0cf1abedbdd49b5ca945ad31a334deeffa4"
Nov 02 01:57:56.999004 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:57:56.998975    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"509e5d228c9c4555f90288f5d15fd0cf1abedbdd49b5ca945ad31a334deeffa4\": container with ID starting with 509e5d228c9c4555f90288f5d15fd0cf1abedbdd49b5ca945ad31a334deeffa4 not found: ID does not exist" containerID="509e5d228c9c4555f90288f5d15fd0cf1abedbdd49b5ca945ad31a334deeffa4"
Nov 02 01:57:56.999106 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.999011    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"509e5d228c9c4555f90288f5d15fd0cf1abedbdd49b5ca945ad31a334deeffa4"} err="failed to get container status \"509e5d228c9c4555f90288f5d15fd0cf1abedbdd49b5ca945ad31a334deeffa4\": rpc error: code = NotFound desc = could not find container \"509e5d228c9c4555f90288f5d15fd0cf1abedbdd49b5ca945ad31a334deeffa4\": container with ID starting with 509e5d228c9c4555f90288f5d15fd0cf1abedbdd49b5ca945ad31a334deeffa4 not found: ID does not exist"
Nov 02 01:57:56.999106 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.999029    5875 scope.go:122] "RemoveContainer" containerID="f8ab5f234644ab5b25f9760f9c8c0c228d4d09595ea95ff6cd8ef737e7e31fa5"
Nov 02 01:57:56.999337 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:57:56.999317    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8ab5f234644ab5b25f9760f9c8c0c228d4d09595ea95ff6cd8ef737e7e31fa5\": container with ID starting with f8ab5f234644ab5b25f9760f9c8c0c228d4d09595ea95ff6cd8ef737e7e31fa5 not found: ID does not exist" containerID="f8ab5f234644ab5b25f9760f9c8c0c228d4d09595ea95ff6cd8ef737e7e31fa5"
Nov 02 01:57:56.999385 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:56.999342    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ab5f234644ab5b25f9760f9c8c0c228d4d09595ea95ff6cd8ef737e7e31fa5"} err="failed to get container status \"f8ab5f234644ab5b25f9760f9c8c0c228d4d09595ea95ff6cd8ef737e7e31fa5\": rpc error: code = NotFound desc = could not find container \"f8ab5f234644ab5b25f9760f9c8c0c228d4d09595ea95ff6cd8ef737e7e31fa5\": container with ID starting with f8ab5f234644ab5b25f9760f9c8c0c228d4d09595ea95ff6cd8ef737e7e31fa5 not found: ID does not exist"
Nov 02 01:57:57.144348 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:57.144316    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"]
Nov 02 01:57:57.201553 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:57.200436    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-b9a2fcb6-gw-openshift-default-765f8d6677-jcstf"]
Nov 02 01:57:58.696447 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:57:58.696419    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="013af201-2be2-4d4a-b07f-edbfd3ec1ac5" path="/var/lib/kubelet/pods/013af201-2be2-4d4a-b07f-edbfd3ec1ac5/volumes"
Nov 02 01:58:10.379892 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:10.379810    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:58:11.319658 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:58:11.319613    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:58:14.595376 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:14.595331    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ts98c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:58:14.595888 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:14.595427    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c" podUID="cf75023b-badb-418a-a93f-48c3d7b0e562" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:58:14.623478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:14.623339    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-mfmmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:58:14.623478 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:14.623410    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj" podUID="004e401a-4d50-46ec-b1bd-d9896f9b20d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:58:16.050249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:16.050201    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e89db2e-tdz2w"]
Nov 02 01:58:16.099554 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:16.099526    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e89db2e-tdz2w"]
Nov 02 01:58:16.696780 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:16.696740    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7447dc9-324a-4f3e-ba29-307ab9e317c4" path="/var/lib/kubelet/pods/f7447dc9-324a-4f3e-ba29-307ab9e317c4/volumes"
Nov 02 01:58:21.131253 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:21.124512    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e74ca69-kvd8n"]
Nov 02 01:58:21.143502 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:21.143465    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e74ca69-kvd8n"]
Nov 02 01:58:22.696355 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:22.696320    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11b0b310-b1b9-420f-bc26-f81d2f3f69ef" path="/var/lib/kubelet/pods/11b0b310-b1b9-420f-bc26-f81d2f3f69ef/volumes"
Nov 02 01:58:25.464473 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:25.464433    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2"]
Nov 02 01:58:25.465382 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:25.465357    5875 cpu_manager.go:414] "containerMap: removing container" podUID="013af201-2be2-4d4a-b07f-edbfd3ec1ac5" containerName="istio-proxy"
Nov 02 01:58:25.465382 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:25.465377    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="013af201-2be2-4d4a-b07f-edbfd3ec1ac5" containerName="istio-proxy"
Nov 02 01:58:25.465534 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:25.465416    5875 cpu_manager.go:414] "containerMap: removing container" podUID="a47b2793-7e4c-4cae-8d55-6bc82c1e4006" containerName="istio-proxy"
Nov 02 01:58:25.465534 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:25.465422    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47b2793-7e4c-4cae-8d55-6bc82c1e4006" containerName="istio-proxy"
Nov 02 01:58:25.465534 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:25.465427    5875 cpu_manager.go:414] "containerMap: removing container" podUID="a47b2793-7e4c-4cae-8d55-6bc82c1e4006" containerName="istio-proxy"
Nov 02 01:58:25.465534 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:25.465433    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47b2793-7e4c-4cae-8d55-6bc82c1e4006" containerName="istio-proxy"
Nov 02 01:58:25.465658 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:25.465556    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47b2793-7e4c-4cae-8d55-6bc82c1e4006" containerName="istio-proxy"
Nov 02 01:58:25.465658 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:25.465566    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47b2793-7e4c-4cae-8d55-6bc82c1e4006" containerName="istio-proxy"
Nov 02 01:58:25.465658 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:25.465608    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="013af201-2be2-4d4a-b07f-edbfd3ec1ac5" containerName="istio-proxy"
Nov 02 01:58:25.465658 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:25.465616    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="013af201-2be2-4d4a-b07f-edbfd3ec1ac5" containerName="istio-proxy"
Nov 02 01:58:25.469959 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:25.469939    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2"
Nov 02 01:58:25.526877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:25.526843    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2"]
Nov 02 01:58:25.571077 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:25.571048    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4f2f926d-serving-cert\""
Nov 02 01:58:25.578273 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:25.578202    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ea638a-9657-45c0-905a-c04ef3d7358f-metrics-certs\") pod \"maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2\" (UID: \"26ea638a-9657-45c0-905a-c04ef3d7358f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2"
Nov 02 01:58:25.578432 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:25.578393    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ea638a-9657-45c0-905a-c04ef3d7358f-maas-api-tls\") pod \"maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2\" (UID: \"26ea638a-9657-45c0-905a-c04ef3d7358f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2"
Nov 02 01:58:25.578485 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:25.578469    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7nxp\" (UniqueName: \"kubernetes.io/projected/26ea638a-9657-45c0-905a-c04ef3d7358f-kube-api-access-w7nxp\") pod \"maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2\" (UID: \"26ea638a-9657-45c0-905a-c04ef3d7358f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2"
Nov 02 01:58:25.578663 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:25.578643    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ea638a-9657-45c0-905a-c04ef3d7358f-openshift-service-ca\") pod \"maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2\" (UID: \"26ea638a-9657-45c0-905a-c04ef3d7358f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2"
Nov 02 01:58:25.679955 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:25.679918    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7nxp\" (UniqueName: \"kubernetes.io/projected/26ea638a-9657-45c0-905a-c04ef3d7358f-kube-api-access-w7nxp\") pod \"maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2\" (UID: \"26ea638a-9657-45c0-905a-c04ef3d7358f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2"
Nov 02 01:58:25.680136 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:25.680027    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ea638a-9657-45c0-905a-c04ef3d7358f-openshift-service-ca\") pod \"maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2\" (UID: \"26ea638a-9657-45c0-905a-c04ef3d7358f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2"
Nov 02 01:58:25.680178 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:25.680141    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ea638a-9657-45c0-905a-c04ef3d7358f-metrics-certs\") pod \"maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2\" (UID: \"26ea638a-9657-45c0-905a-c04ef3d7358f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2"
Nov 02 01:58:25.680256 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:25.680211    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ea638a-9657-45c0-905a-c04ef3d7358f-maas-api-tls\") pod \"maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2\" (UID: \"26ea638a-9657-45c0-905a-c04ef3d7358f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2"
Nov 02 01:58:25.680407 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:58:25.680387    5875 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-serving-cert: secret "maas-api-e2e-ait-4f2f926d-serving-cert" not found
Nov 02 01:58:25.680470 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:58:25.680460    5875 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ea638a-9657-45c0-905a-c04ef3d7358f-maas-api-tls podName:26ea638a-9657-45c0-905a-c04ef3d7358f nodeName:}" failed. No retries permitted until 2026-11-02 01:58:26.180445964 +0000 UTC m=+1553.621482443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/26ea638a-9657-45c0-905a-c04ef3d7358f-maas-api-tls") pod "maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2" (UID: "26ea638a-9657-45c0-905a-c04ef3d7358f") : secret "maas-api-e2e-ait-4f2f926d-serving-cert" not found
Nov 02 01:58:25.680800 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:25.680774    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ea638a-9657-45c0-905a-c04ef3d7358f-openshift-service-ca\") pod \"maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2\" (UID: \"26ea638a-9657-45c0-905a-c04ef3d7358f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2"
Nov 02 01:58:25.682828 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:25.682805    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ea638a-9657-45c0-905a-c04ef3d7358f-metrics-certs\") pod \"maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2\" (UID: \"26ea638a-9657-45c0-905a-c04ef3d7358f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2"
Nov 02 01:58:25.767308 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:25.752688    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7nxp\" (UniqueName: \"kubernetes.io/projected/26ea638a-9657-45c0-905a-c04ef3d7358f-kube-api-access-w7nxp\") pod \"maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2\" (UID: \"26ea638a-9657-45c0-905a-c04ef3d7358f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2"
Nov 02 01:58:26.206396 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.206363    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ea638a-9657-45c0-905a-c04ef3d7358f-maas-api-tls\") pod \"maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2\" (UID: \"26ea638a-9657-45c0-905a-c04ef3d7358f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2"
Nov 02 01:58:26.231171 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.229666    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ea638a-9657-45c0-905a-c04ef3d7358f-maas-api-tls\") pod \"maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2\" (UID: \"26ea638a-9657-45c0-905a-c04ef3d7358f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2"
Nov 02 01:58:26.244726 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.244688    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"]
Nov 02 01:58:26.245897 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.245871    5875 cpu_manager.go:414] "containerMap: removing container" podUID="013af201-2be2-4d4a-b07f-edbfd3ec1ac5" containerName="istio-proxy"
Nov 02 01:58:26.245897 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.245893    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="013af201-2be2-4d4a-b07f-edbfd3ec1ac5" containerName="istio-proxy"
Nov 02 01:58:26.252387 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.251161    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.327373 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.327342    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"]
Nov 02 01:58:26.339315 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.339288    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-73e1cc-openshift-default-dockercfg-bdrb6\""
Nov 02 01:58:26.393252 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.392507    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2"
Nov 02 01:58:26.417249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.412353    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-workload-certs\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.417249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.412420    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-token\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.417249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.412450    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgcd4\" (UniqueName: \"kubernetes.io/projected/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-kube-api-access-xgcd4\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.417249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.412511    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istiod-ca-cert\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.417249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.412541    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-data\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.417249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.412620    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-credential-socket\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.417249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.412682    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-podinfo\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.417249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.412865    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-workload-socket\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.417249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.413013    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-envoy\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.534072 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.529541    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-credential-socket\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.534072 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.529622    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-podinfo\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.534072 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.529782    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-workload-socket\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.534072 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.529906    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-envoy\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.534072 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.529985    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-workload-certs\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.534072 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.530029    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-token\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.534072 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.530055    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgcd4\" (UniqueName: \"kubernetes.io/projected/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-kube-api-access-xgcd4\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.534072 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.530114    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istiod-ca-cert\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.534072 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.530146    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-data\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.534072 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.530715    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-data\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.534072 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.530993    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-credential-socket\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.549018 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.548724    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-workload-socket\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.549645 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.549610    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-workload-certs\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.549817 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.549751    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istiod-ca-cert\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.556471 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.556414    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-envoy\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.558154 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.557822    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-podinfo\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.614070 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.614019    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-token\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.618270 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.617818    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgcd4\" (UniqueName: \"kubernetes.io/projected/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-kube-api-access-xgcd4\") pod \"e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") " pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.872390 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.872308    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:26.927838 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.927806    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2"]
Nov 02 01:58:26.928140 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:58:26.928111    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ea638a_9657_45c0_905a_c04ef3d7358f.slice/crio-f162af31163eb580a10ffbe6abf9dd2761b2681888ae070c98a3278a0ecec279 WatchSource:0}: Error finding container f162af31163eb580a10ffbe6abf9dd2761b2681888ae070c98a3278a0ecec279: Status 404 returned error can't find the container with id f162af31163eb580a10ffbe6abf9dd2761b2681888ae070c98a3278a0ecec279
Nov 02 01:58:26.929765 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:26.929744    5875 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:58:27.268969 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:58:27.268941    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b7a180_01b7_4c32_93be_dcb96b6b3b42.slice/crio-a60e052c328432e3bf3593d373f27b9f11a2da2359e3c9bb309d2f9cbb238879 WatchSource:0}: Error finding container a60e052c328432e3bf3593d373f27b9f11a2da2359e3c9bb309d2f9cbb238879: Status 404 returned error can't find the container with id a60e052c328432e3bf3593d373f27b9f11a2da2359e3c9bb309d2f9cbb238879
Nov 02 01:58:27.269190 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:27.269168    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"]
Nov 02 01:58:27.272231 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:27.272181    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:58:27.272330 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:27.272293    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:58:27.272372 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:27.272328    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:58:27.322238 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:27.322190    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf" event={"ID":"d4b7a180-01b7-4c32-93be-dcb96b6b3b42","Type":"ContainerStarted","Data":"a60e052c328432e3bf3593d373f27b9f11a2da2359e3c9bb309d2f9cbb238879"}
Nov 02 01:58:27.323585 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:27.323558    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2" event={"ID":"26ea638a-9657-45c0-905a-c04ef3d7358f","Type":"ContainerStarted","Data":"f162af31163eb580a10ffbe6abf9dd2761b2681888ae070c98a3278a0ecec279"}
Nov 02 01:58:28.334379 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:28.334343    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf" event={"ID":"d4b7a180-01b7-4c32-93be-dcb96b6b3b42","Type":"ContainerStarted","Data":"639b38cd3662735e99dce69faf3e49b82f5b265b24507010e1ef42dc813016a9"}
Nov 02 01:58:28.336231 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:28.336194    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2" event={"ID":"26ea638a-9657-45c0-905a-c04ef3d7358f","Type":"ContainerStarted","Data":"7c17640f22e25649cc7a26ab008dcb674b21fcbe1d0684b1635f0871990548a7"}
Nov 02 01:58:28.336347 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:28.336290    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2"
Nov 02 01:58:28.504315 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:28.504215    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf" podStartSLOduration=2.504194177 podStartE2EDuration="2.504194177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:28.502463895 +0000 UTC m=+1555.943500403" watchObservedRunningTime="2026-11-02 01:58:28.504194177 +0000 UTC m=+1555.945230675"
Nov 02 01:58:28.623565 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:28.623466    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2" podStartSLOduration=2.706153578 podStartE2EDuration="3.623452997s" totalImagesPullingTime="917.299419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:28.619474782 +0000 UTC m=+1556.060511292" watchObservedRunningTime="2026-11-02 01:58:28.623452997 +0000 UTC m=+1556.064489494"
Nov 02 01:58:28.875775 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:28.875593    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:28.883341 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:28.883312    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:29.344414 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:29.344375    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:29.345281 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:29.345262    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 01:58:34.324909 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.324877    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"]
Nov 02 01:58:34.329664 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.329645    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.349792 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.349767    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2"
Nov 02 01:58:34.409931 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.400507    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-25789ee1-openshift-default-dockercfg-w6dqt\""
Nov 02 01:58:34.423282 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.423243    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-credential-socket\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.423541 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.423520    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eedec700-8816-4b96-a087-7d0d691123bf-istio-podinfo\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.423773 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.423756    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eedec700-8816-4b96-a087-7d0d691123bf-istio-token\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.423949 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.423934    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eedec700-8816-4b96-a087-7d0d691123bf-istiod-ca-cert\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.424138 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.424123    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-istio-envoy\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.585927 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.585834    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-workload-socket\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.586192 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.586154    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9px6\" (UniqueName: \"kubernetes.io/projected/eedec700-8816-4b96-a087-7d0d691123bf-kube-api-access-j9px6\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.586409 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.586386    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-workload-certs\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.586606 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.586590    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-istio-data\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.615344 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.615305    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"]
Nov 02 01:58:34.760637 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.760586    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-workload-socket\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.760916 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.760893    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9px6\" (UniqueName: \"kubernetes.io/projected/eedec700-8816-4b96-a087-7d0d691123bf-kube-api-access-j9px6\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.761134 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.761115    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-workload-certs\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.761325 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.761307    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-istio-data\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.761503 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.761482    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-credential-socket\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.761705 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.761662    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eedec700-8816-4b96-a087-7d0d691123bf-istio-podinfo\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.761947 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.761929    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eedec700-8816-4b96-a087-7d0d691123bf-istio-token\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.762124 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.762108    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eedec700-8816-4b96-a087-7d0d691123bf-istiod-ca-cert\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.762310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.762295    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-istio-envoy\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.775255 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.773237    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-workload-socket\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.808274 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.796257    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-credential-socket\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.954374 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.812956    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eedec700-8816-4b96-a087-7d0d691123bf-istio-podinfo\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.954613 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.821407    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-workload-certs\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.968298 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.821641    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-istio-data\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.968496 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.827820    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eedec700-8816-4b96-a087-7d0d691123bf-istiod-ca-cert\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.968573 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.942924    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9px6\" (UniqueName: \"kubernetes.io/projected/eedec700-8816-4b96-a087-7d0d691123bf-kube-api-access-j9px6\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.974304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.946927    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-istio-envoy\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:34.995286 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:34.990796    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eedec700-8816-4b96-a087-7d0d691123bf-istio-token\") pod \"e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") " pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:35.278925 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:35.278864    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8"]
Nov 02 01:58:35.290253 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:35.282702    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:35.300741 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:35.300711    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8"
Nov 02 01:58:35.396242 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:35.377540    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-73e1cc-dockercfg-z57qx\""
Nov 02 01:58:35.422257 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:35.399197    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-73e1cc\""
Nov 02 01:58:35.478468 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:35.477870    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpf67\" (UniqueName: \"kubernetes.io/projected/8425af9f-79bc-4f19-9109-f6712d7b8198-kube-api-access-lpf67\") pod \"payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8\" (UID: \"8425af9f-79bc-4f19-9109-f6712d7b8198\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8"
Nov 02 01:58:35.478468 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:35.477942    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8425af9f-79bc-4f19-9109-f6712d7b8198-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8\" (UID: \"8425af9f-79bc-4f19-9109-f6712d7b8198\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8"
Nov 02 01:58:35.667464 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:35.667357    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpf67\" (UniqueName: \"kubernetes.io/projected/8425af9f-79bc-4f19-9109-f6712d7b8198-kube-api-access-lpf67\") pod \"payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8\" (UID: \"8425af9f-79bc-4f19-9109-f6712d7b8198\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8"
Nov 02 01:58:35.667778 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:35.667753    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8425af9f-79bc-4f19-9109-f6712d7b8198-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8\" (UID: \"8425af9f-79bc-4f19-9109-f6712d7b8198\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8"
Nov 02 01:58:35.669420 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:35.669383    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8425af9f-79bc-4f19-9109-f6712d7b8198-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8\" (UID: \"8425af9f-79bc-4f19-9109-f6712d7b8198\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8"
Nov 02 01:58:35.689727 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:35.689697    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8"]
Nov 02 01:58:35.747869 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:35.747838    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpf67\" (UniqueName: \"kubernetes.io/projected/8425af9f-79bc-4f19-9109-f6712d7b8198-kube-api-access-lpf67\") pod \"payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8\" (UID: \"8425af9f-79bc-4f19-9109-f6712d7b8198\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8"
Nov 02 01:58:35.973716 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:35.973688    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8"
Nov 02 01:58:36.122296 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:58:36.122209    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeedec700_8816_4b96_a087_7d0d691123bf.slice/crio-da0dcf95c49f05ddd6b2f441f4e84233713ae74e7795a67a23e3f856aa45d7c8 WatchSource:0}: Error finding container da0dcf95c49f05ddd6b2f441f4e84233713ae74e7795a67a23e3f856aa45d7c8: Status 404 returned error can't find the container with id da0dcf95c49f05ddd6b2f441f4e84233713ae74e7795a67a23e3f856aa45d7c8
Nov 02 01:58:36.122513 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:36.122490    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"]
Nov 02 01:58:36.125270 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:36.125240    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:58:36.125341 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:36.125304    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:58:36.125341 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:36.125333    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:58:36.359925 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:58:36.359891    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8425af9f_79bc_4f19_9109_f6712d7b8198.slice/crio-df264f18fe23dcaf0fd9fb0dffb45ce9fe8892849e8e6f58a23e2d3cd8a59c03 WatchSource:0}: Error finding container df264f18fe23dcaf0fd9fb0dffb45ce9fe8892849e8e6f58a23e2d3cd8a59c03: Status 404 returned error can't find the container with id df264f18fe23dcaf0fd9fb0dffb45ce9fe8892849e8e6f58a23e2d3cd8a59c03
Nov 02 01:58:36.360854 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:36.360082    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8"]
Nov 02 01:58:36.826314 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:36.823837    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8"
Nov 02 01:58:36.826314 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:36.823892    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8" event={"ID":"8425af9f-79bc-4f19-9109-f6712d7b8198","Type":"ContainerStarted","Data":"52f34d071b5acd5cfda0b6611b43cf0c2bab1680f506a05e90eb6346b8f8ed3e"}
Nov 02 01:58:36.826314 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:36.823921    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8" event={"ID":"8425af9f-79bc-4f19-9109-f6712d7b8198","Type":"ContainerStarted","Data":"df264f18fe23dcaf0fd9fb0dffb45ce9fe8892849e8e6f58a23e2d3cd8a59c03"}
Nov 02 01:58:36.826314 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:36.823944    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f" event={"ID":"eedec700-8816-4b96-a087-7d0d691123bf","Type":"ContainerStarted","Data":"7473807fe018380eaa349742a6a3139a3dd4262b48a624125925debf92ce53b4"}
Nov 02 01:58:36.826314 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:36.823968    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f" event={"ID":"eedec700-8816-4b96-a087-7d0d691123bf","Type":"ContainerStarted","Data":"da0dcf95c49f05ddd6b2f441f4e84233713ae74e7795a67a23e3f856aa45d7c8"}
Nov 02 01:58:36.933285 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:36.933184    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8" podStartSLOduration=1.933162899 podStartE2EDuration="1.933162899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:36.932998487 +0000 UTC m=+1564.374034994" watchObservedRunningTime="2026-11-02 01:58:36.933162899 +0000 UTC m=+1564.374199397"
Nov 02 01:58:37.093668 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:37.093460    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f" podStartSLOduration=3.09343529 podStartE2EDuration="3.09343529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:37.064788887 +0000 UTC m=+1564.505825432" watchObservedRunningTime="2026-11-02 01:58:37.09343529 +0000 UTC m=+1564.534471793"
Nov 02 01:58:37.294243 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:37.290714    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:37.333076 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:37.333046    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:37.829018 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:37.828941    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:37.829895 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:37.829873    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 01:58:39.048072 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:39.048040    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86a6787c-9xzv2"]
Nov 02 01:58:39.097103 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:39.097078    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86a6787c-9xzv2"]
Nov 02 01:58:40.696282 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:40.696206    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea66e631-971b-41b3-a868-d17e08766620" path="/var/lib/kubelet/pods/ea66e631-971b-41b3-a868-d17e08766620/volumes"
Nov 02 01:58:41.113311 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.113279    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"]
Nov 02 01:58:41.119446 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.119428    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.171466 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.171436    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-606ca0-openshift-default-dockercfg-hsd99\""
Nov 02 01:58:41.227670 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.227636    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"]
Nov 02 01:58:41.320253 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.312657    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-envoy\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.320253 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.312733    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-podinfo\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.320253 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.312838    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-workload-certs\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.320253 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.312918    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-data\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.320253 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.313349    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-token\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.320253 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.313447    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-workload-socket\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.320253 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.313555    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a054e97-d2a9-47fd-a380-24f4c97d908e-istiod-ca-cert\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.320253 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.313582    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-credential-socket\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.320253 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.313641    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsm2p\" (UniqueName: \"kubernetes.io/projected/9a054e97-d2a9-47fd-a380-24f4c97d908e-kube-api-access-wsm2p\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.442767 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.440739    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-envoy\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.442767 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.440804    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-podinfo\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.442767 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.440899    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-workload-certs\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.442767 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.440971    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-data\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.442767 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.441121    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-token\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.442767 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.441176    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-workload-socket\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.442767 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.441285    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a054e97-d2a9-47fd-a380-24f4c97d908e-istiod-ca-cert\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.442767 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.441311    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-credential-socket\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.442767 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.441368    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsm2p\" (UniqueName: \"kubernetes.io/projected/9a054e97-d2a9-47fd-a380-24f4c97d908e-kube-api-access-wsm2p\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.452504 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.452469    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-workload-certs\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.453050 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.453008    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-data\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.453984 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.453964    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a054e97-d2a9-47fd-a380-24f4c97d908e-istiod-ca-cert\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.454491 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.454446    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-workload-socket\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.454859 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.454841    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-credential-socket\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.469441 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.469408    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-envoy\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.472018 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:58:41.471894    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:58:41.473342 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.473317    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-podinfo\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.502628 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.502595    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsm2p\" (UniqueName: \"kubernetes.io/projected/9a054e97-d2a9-47fd-a380-24f4c97d908e-kube-api-access-wsm2p\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.503881 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.503856    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-token\") pod \"e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") " pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:41.735004 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:41.734970    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:42.168516 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:42.168486    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"]
Nov 02 01:58:42.169823 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:58:42.169797    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a054e97_d2a9_47fd_a380_24f4c97d908e.slice/crio-0c14353e823619ae7e46250b799a631bf6dbcc75938f5021019dcd300f5a1622 WatchSource:0}: Error finding container 0c14353e823619ae7e46250b799a631bf6dbcc75938f5021019dcd300f5a1622: Status 404 returned error can't find the container with id 0c14353e823619ae7e46250b799a631bf6dbcc75938f5021019dcd300f5a1622
Nov 02 01:58:42.172559 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:42.172528    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:58:42.172660 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:42.172592    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:58:42.172660 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:42.172620    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:58:42.956533 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:42.956493    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg" event={"ID":"9a054e97-d2a9-47fd-a380-24f4c97d908e","Type":"ContainerStarted","Data":"19244c9bd784f673fa14b8bef85d37ceeefef9c38799461dfde948ae773e2ee0"}
Nov 02 01:58:42.957040 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:42.956551    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg" event={"ID":"9a054e97-d2a9-47fd-a380-24f4c97d908e","Type":"ContainerStarted","Data":"0c14353e823619ae7e46250b799a631bf6dbcc75938f5021019dcd300f5a1622"}
Nov 02 01:58:43.128197 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:43.128137    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg" podStartSLOduration=2.128123109 podStartE2EDuration="2.128123109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:43.126214374 +0000 UTC m=+1570.567250872" watchObservedRunningTime="2026-11-02 01:58:43.128123109 +0000 UTC m=+1570.569159605"
Nov 02 01:58:43.735581 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:43.735549    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:43.741578 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:43.741552    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:43.966182 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:43.966150    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:43.966939 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:43.966918    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 01:58:44.051032 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:44.050944    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b6d0e87-w5t5m"]
Nov 02 01:58:44.058087 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:44.058058    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b6d0e87-w5t5m"]
Nov 02 01:58:44.697235 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:44.697190    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6" path="/var/lib/kubelet/pods/19b0eeb4-a43a-48ac-9f2b-ff2c267c6ca6/volumes"
Nov 02 01:58:47.832349 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:47.832319    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8"
Nov 02 01:58:51.656487 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:51.655914    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf"]
Nov 02 01:58:51.666012 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:51.665099    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf"
Nov 02 01:58:51.694079 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:51.694045    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf"]
Nov 02 01:58:51.722249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:51.720501    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-606ca0\""
Nov 02 01:58:51.722249 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:51.720695    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-606ca0-dockercfg-xpz5j\""
Nov 02 01:58:51.776763 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:51.776722    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72076bb7-2227-495a-a707-5bfba8979668-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf\" (UID: \"72076bb7-2227-495a-a707-5bfba8979668\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf"
Nov 02 01:58:51.777049 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:51.777025    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvjxs\" (UniqueName: \"kubernetes.io/projected/72076bb7-2227-495a-a707-5bfba8979668-kube-api-access-bvjxs\") pod \"payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf\" (UID: \"72076bb7-2227-495a-a707-5bfba8979668\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf"
Nov 02 01:58:51.886667 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:51.885587    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72076bb7-2227-495a-a707-5bfba8979668-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf\" (UID: \"72076bb7-2227-495a-a707-5bfba8979668\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf"
Nov 02 01:58:51.886667 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:51.885655    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvjxs\" (UniqueName: \"kubernetes.io/projected/72076bb7-2227-495a-a707-5bfba8979668-kube-api-access-bvjxs\") pod \"payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf\" (UID: \"72076bb7-2227-495a-a707-5bfba8979668\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf"
Nov 02 01:58:51.886667 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:51.886602    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72076bb7-2227-495a-a707-5bfba8979668-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf\" (UID: \"72076bb7-2227-495a-a707-5bfba8979668\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf"
Nov 02 01:58:51.955468 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:51.955436    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvjxs\" (UniqueName: \"kubernetes.io/projected/72076bb7-2227-495a-a707-5bfba8979668-kube-api-access-bvjxs\") pod \"payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf\" (UID: \"72076bb7-2227-495a-a707-5bfba8979668\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf"
Nov 02 01:58:52.015515 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:52.015478    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf"
Nov 02 01:58:52.405002 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:58:52.404975    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72076bb7_2227_495a_a707_5bfba8979668.slice/crio-26c01d1e33ca082720be2801e61ce928466d4bbaf3fda8d992fab321202825d7 WatchSource:0}: Error finding container 26c01d1e33ca082720be2801e61ce928466d4bbaf3fda8d992fab321202825d7: Status 404 returned error can't find the container with id 26c01d1e33ca082720be2801e61ce928466d4bbaf3fda8d992fab321202825d7
Nov 02 01:58:52.405155 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:52.405079    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf"]
Nov 02 01:58:52.436430 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:52.436390    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf" event={"ID":"72076bb7-2227-495a-a707-5bfba8979668","Type":"ContainerStarted","Data":"26c01d1e33ca082720be2801e61ce928466d4bbaf3fda8d992fab321202825d7"}
Nov 02 01:58:53.072147 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:53.072116    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-472402a9-7dwkg"]
Nov 02 01:58:53.103259 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:53.103205    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-472402a9-7dwkg"]
Nov 02 01:58:53.447722 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:53.447622    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf" event={"ID":"72076bb7-2227-495a-a707-5bfba8979668","Type":"ContainerStarted","Data":"13ee2e1b3a690d5404a86eaeb3124181212a4b395330556bf1f7d49e89a0f75a"}
Nov 02 01:58:53.447722 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:53.447706    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf"
Nov 02 01:58:53.564400 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:53.564335    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf" podStartSLOduration=2.564319704 podStartE2EDuration="2.564319704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:53.563588732 +0000 UTC m=+1581.004625241" watchObservedRunningTime="2026-11-02 01:58:53.564319704 +0000 UTC m=+1581.005356201"
Nov 02 01:58:54.699309 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:54.699274    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6e8666e-3f83-4c1b-ae4c-5d76eb70069c" path="/var/lib/kubelet/pods/b6e8666e-3f83-4c1b-ae4c-5d76eb70069c/volumes"
Nov 02 01:58:56.279347 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:56.279310    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d8d7057-w7tml"]
Nov 02 01:58:56.286040 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:56.286017    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8d7057-w7tml"
Nov 02 01:58:56.335998 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:56.335971    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d8d7057-w7tml"]
Nov 02 01:58:56.379816 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:56.379785    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:58:56.379978 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:56.379922    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:58:56.385281 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:56.385244    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bdf2f06-8069-472f-81f1-7def8ca7cb27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d8d7057-w7tml\" (UID: \"2bdf2f06-8069-472f-81f1-7def8ca7cb27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8d7057-w7tml"
Nov 02 01:58:56.385474 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:56.385456    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bdf2f06-8069-472f-81f1-7def8ca7cb27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d8d7057-w7tml\" (UID: \"2bdf2f06-8069-472f-81f1-7def8ca7cb27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8d7057-w7tml"
Nov 02 01:58:56.385603 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:56.385588    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnfmk\" (UniqueName: \"kubernetes.io/projected/2bdf2f06-8069-472f-81f1-7def8ca7cb27-kube-api-access-hnfmk\") pod \"maas-api-revoke-sub-7d8d7057-w7tml\" (UID: \"2bdf2f06-8069-472f-81f1-7def8ca7cb27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8d7057-w7tml"
Nov 02 01:58:56.473508 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:56.473477    5875 generic.go:356] "Generic (PLEG): container finished" podID="98296768-250b-4160-8678-b92e2fb3a151" containerID="22a9264bd402ba32e032bfc65f004c848e702ebac53f1a5c3f974db876bf995f" exitCode=0
Nov 02 01:58:56.473686 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:56.473556    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz" event={"ID":"98296768-250b-4160-8678-b92e2fb3a151","Type":"ContainerDied","Data":"22a9264bd402ba32e032bfc65f004c848e702ebac53f1a5c3f974db876bf995f"}
Nov 02 01:58:56.474034 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:56.474014    5875 scope.go:122] "RemoveContainer" containerID="22a9264bd402ba32e032bfc65f004c848e702ebac53f1a5c3f974db876bf995f"
Nov 02 01:58:56.475704 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:56.475673    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:58:56.475793 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:56.475743    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:58:56.475793 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:56.475776    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:58:56.487555 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:56.487531    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bdf2f06-8069-472f-81f1-7def8ca7cb27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d8d7057-w7tml\" (UID: \"2bdf2f06-8069-472f-81f1-7def8ca7cb27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8d7057-w7tml"
Nov 02 01:58:56.487678 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:56.487663    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnfmk\" (UniqueName: \"kubernetes.io/projected/2bdf2f06-8069-472f-81f1-7def8ca7cb27-kube-api-access-hnfmk\") pod \"maas-api-revoke-sub-7d8d7057-w7tml\" (UID: \"2bdf2f06-8069-472f-81f1-7def8ca7cb27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8d7057-w7tml"
Nov 02 01:58:56.487836 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:56.487819    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bdf2f06-8069-472f-81f1-7def8ca7cb27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d8d7057-w7tml\" (UID: \"2bdf2f06-8069-472f-81f1-7def8ca7cb27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8d7057-w7tml"
Nov 02 01:58:56.488265 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:56.488246    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bdf2f06-8069-472f-81f1-7def8ca7cb27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d8d7057-w7tml\" (UID: \"2bdf2f06-8069-472f-81f1-7def8ca7cb27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8d7057-w7tml"
Nov 02 01:58:56.490481 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:56.490463    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:58:56.490769 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:56.490752    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bdf2f06-8069-472f-81f1-7def8ca7cb27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d8d7057-w7tml\" (UID: \"2bdf2f06-8069-472f-81f1-7def8ca7cb27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8d7057-w7tml"
Nov 02 01:58:56.548760 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:56.548736    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnfmk\" (UniqueName: \"kubernetes.io/projected/2bdf2f06-8069-472f-81f1-7def8ca7cb27-kube-api-access-hnfmk\") pod \"maas-api-revoke-sub-7d8d7057-w7tml\" (UID: \"2bdf2f06-8069-472f-81f1-7def8ca7cb27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8d7057-w7tml"
Nov 02 01:58:56.601073 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:56.601042    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8d7057-w7tml"
Nov 02 01:58:57.033540 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:57.033511    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d8d7057-w7tml"]
Nov 02 01:58:57.033766 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:58:57.033745    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bdf2f06_8069_472f_81f1_7def8ca7cb27.slice/crio-e0553ddd5fcb32c2bd01bada9c065edee366d3e92400ad0e1f1be099b0d5c913 WatchSource:0}: Error finding container e0553ddd5fcb32c2bd01bada9c065edee366d3e92400ad0e1f1be099b0d5c913: Status 404 returned error can't find the container with id e0553ddd5fcb32c2bd01bada9c065edee366d3e92400ad0e1f1be099b0d5c913
Nov 02 01:58:57.483748 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:57.483688    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8d7057-w7tml" event={"ID":"2bdf2f06-8069-472f-81f1-7def8ca7cb27","Type":"ContainerStarted","Data":"aa0fed67b6e51dcf5abbb897243696b1eb384b1ab3ff5a5b4f04e311e0e50208"}
Nov 02 01:58:57.484253 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:57.483769    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8d7057-w7tml" event={"ID":"2bdf2f06-8069-472f-81f1-7def8ca7cb27","Type":"ContainerStarted","Data":"e0553ddd5fcb32c2bd01bada9c065edee366d3e92400ad0e1f1be099b0d5c913"}
Nov 02 01:58:57.485877 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:57.485850    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz" event={"ID":"98296768-250b-4160-8678-b92e2fb3a151","Type":"ContainerStarted","Data":"cd64b3f296c66d4c5bc3bd3c63362e50d740a4b6b8c0e63fd116ee4500f86f80"}
Nov 02 01:58:57.547891 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:57.547858    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:58:57.549017 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:57.548991    5875 patch_prober.go:29] interesting pod/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 01:58:57.549150 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:57.549039    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz" podUID="98296768-250b-4160-8678-b92e2fb3a151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 01:58:57.626865 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:57.626797    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8d7057-w7tml" podStartSLOduration=1.6267764 podStartE2EDuration="1.6267764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:57.601633137 +0000 UTC m=+1585.042669636" watchObservedRunningTime="2026-11-02 01:58:57.6267764 +0000 UTC m=+1585.067812898"
Nov 02 01:58:58.495832 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:58.495798    5875 generic.go:356] "Generic (PLEG): container finished" podID="2bdf2f06-8069-472f-81f1-7def8ca7cb27" containerID="aa0fed67b6e51dcf5abbb897243696b1eb384b1ab3ff5a5b4f04e311e0e50208" exitCode=0
Nov 02 01:58:58.496300 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:58.495895    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8d7057-w7tml" event={"ID":"2bdf2f06-8069-472f-81f1-7def8ca7cb27","Type":"ContainerDied","Data":"aa0fed67b6e51dcf5abbb897243696b1eb384b1ab3ff5a5b4f04e311e0e50208"}
Nov 02 01:58:58.547957 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:58.547924    5875 patch_prober.go:29] interesting pod/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 01:58:58.548136 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:58.547977    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz" podUID="98296768-250b-4160-8678-b92e2fb3a151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 01:58:59.101246 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:59.099796    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07650e93-llw4h"]
Nov 02 01:58:59.101246 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:59.099868    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07650e93-llw4h"]
Nov 02 01:58:59.547939 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:59.547902    5875 patch_prober.go:29] interesting pod/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 01:58:59.548372 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:59.547986    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz" podUID="98296768-250b-4160-8678-b92e2fb3a151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 01:58:59.780932 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:59.780907    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8d7057-w7tml"
Nov 02 01:58:59.831591 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:59.831511    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bdf2f06-8069-472f-81f1-7def8ca7cb27-kube-api-access-hnfmk\" (UniqueName: \"kubernetes.io/projected/2bdf2f06-8069-472f-81f1-7def8ca7cb27-kube-api-access-hnfmk\") pod \"2bdf2f06-8069-472f-81f1-7def8ca7cb27\" (UID: \"2bdf2f06-8069-472f-81f1-7def8ca7cb27\") "
Nov 02 01:58:59.831591 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:59.831570    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bdf2f06-8069-472f-81f1-7def8ca7cb27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bdf2f06-8069-472f-81f1-7def8ca7cb27-maas-api-openshift-ca\") pod \"2bdf2f06-8069-472f-81f1-7def8ca7cb27\" (UID: \"2bdf2f06-8069-472f-81f1-7def8ca7cb27\") "
Nov 02 01:58:59.831919 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:59.831888    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bdf2f06-8069-472f-81f1-7def8ca7cb27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bdf2f06-8069-472f-81f1-7def8ca7cb27-maas-api-xks-ca\") pod \"2bdf2f06-8069-472f-81f1-7def8ca7cb27\" (UID: \"2bdf2f06-8069-472f-81f1-7def8ca7cb27\") "
Nov 02 01:58:59.832064 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:59.831941    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdf2f06-8069-472f-81f1-7def8ca7cb27-maas-api-openshift-ca" pod "2bdf2f06-8069-472f-81f1-7def8ca7cb27" (UID: "2bdf2f06-8069-472f-81f1-7def8ca7cb27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:58:59.832544 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:59.832454    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bdf2f06-8069-472f-81f1-7def8ca7cb27-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:58:59.833954 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:59.833930    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bdf2f06-8069-472f-81f1-7def8ca7cb27-kube-api-access-hnfmk" pod "2bdf2f06-8069-472f-81f1-7def8ca7cb27" (UID: "2bdf2f06-8069-472f-81f1-7def8ca7cb27"). InnerVolumeSpecName "kube-api-access-hnfmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:58:59.834063 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:59.833958    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bdf2f06-8069-472f-81f1-7def8ca7cb27-maas-api-xks-ca" pod "2bdf2f06-8069-472f-81f1-7def8ca7cb27" (UID: "2bdf2f06-8069-472f-81f1-7def8ca7cb27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:58:59.934042 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:59.934005    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bdf2f06-8069-472f-81f1-7def8ca7cb27-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:58:59.934042 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:58:59.934047    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnfmk\" (UniqueName: \"kubernetes.io/projected/2bdf2f06-8069-472f-81f1-7def8ca7cb27-kube-api-access-hnfmk\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:00.540231 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:00.540193    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8d7057-w7tml" event={"ID":"2bdf2f06-8069-472f-81f1-7def8ca7cb27","Type":"ContainerDied","Data":"e0553ddd5fcb32c2bd01bada9c065edee366d3e92400ad0e1f1be099b0d5c913"}
Nov 02 01:59:00.540423 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:00.540273    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0553ddd5fcb32c2bd01bada9c065edee366d3e92400ad0e1f1be099b0d5c913"
Nov 02 01:59:00.540423 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:00.540206    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8d7057-w7tml"
Nov 02 01:59:00.548112 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:00.548085    5875 patch_prober.go:29] interesting pod/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 01:59:00.548483 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:00.548126    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz" podUID="98296768-250b-4160-8678-b92e2fb3a151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 01:59:00.696604 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:00.696574    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7467eb0a-7afd-478f-94a7-95d7f22564f6" path="/var/lib/kubelet/pods/7467eb0a-7afd-478f-94a7-95d7f22564f6/volumes"
Nov 02 01:59:01.547919 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:01.547883    5875 patch_prober.go:29] interesting pod/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 01:59:01.548110 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:01.547946    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz" podUID="98296768-250b-4160-8678-b92e2fb3a151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 01:59:02.548070 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:02.548038    5875 patch_prober.go:29] interesting pod/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 01:59:02.548481 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:02.548105    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz" podUID="98296768-250b-4160-8678-b92e2fb3a151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 01:59:03.051729 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:03.051693    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99e23104-7b8vf"]
Nov 02 01:59:03.100487 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:03.100453    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99e23104-7b8vf"]
Nov 02 01:59:03.552344 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:03.552310    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:59:03.552833 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:03.552754    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:59:03.553650 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:03.553628    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 01:59:04.459413 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:04.459383    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf"
Nov 02 01:59:04.714631 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:04.714552    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a" path="/var/lib/kubelet/pods/ee71483c-13b2-4ac9-9b3c-8f8bf8b6024a/volumes"
Nov 02 01:59:11.320148 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:59:11.320109    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:59:12.048762 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:12.048735    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5328a392-lz8ws"]
Nov 02 01:59:12.098317 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:12.098293    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5328a392-lz8ws"]
Nov 02 01:59:12.698668 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:12.698637    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f449c9d-4431-40de-9f2e-4299fa39cb37" path="/var/lib/kubelet/pods/4f449c9d-4431-40de-9f2e-4299fa39cb37/volumes"
Nov 02 01:59:17.048776 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:17.048746    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a4e411c-xhrxx"]
Nov 02 01:59:17.097944 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:17.097912    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a4e411c-xhrxx"]
Nov 02 01:59:18.047847 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:18.047795    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f96dcece-5rkqn"]
Nov 02 01:59:18.051955 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:18.051931    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f96dcece-5rkqn"]
Nov 02 01:59:18.698440 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:18.698411    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9fe9e10-c020-4f72-a108-72cabf18cbae" path="/var/lib/kubelet/pods/b9fe9e10-c020-4f72-a108-72cabf18cbae/volumes"
Nov 02 01:59:18.698880 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:18.698865    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4d60ff0-bf59-4022-a94f-1c3e560a7a6c" path="/var/lib/kubelet/pods/e4d60ff0-bf59-4022-a94f-1c3e560a7a6c/volumes"
Nov 02 01:59:21.409246 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:21.407541    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 01:59:23.019592 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:23.019560    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm"]
Nov 02 01:59:23.021343 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:23.021306    5875 cpu_manager.go:414] "containerMap: removing container" podUID="2bdf2f06-8069-472f-81f1-7def8ca7cb27" containerName="revoke-keys"
Nov 02 01:59:23.021511 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:23.021495    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bdf2f06-8069-472f-81f1-7def8ca7cb27" containerName="revoke-keys"
Nov 02 01:59:23.021889 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:23.021872    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bdf2f06-8069-472f-81f1-7def8ca7cb27" containerName="revoke-keys"
Nov 02 01:59:23.045838 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:23.045808    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm"
Nov 02 01:59:23.103037 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:23.099669    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm"]
Nov 02 01:59:23.118534 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:23.118503    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-73e1cc-serving-cert\""
Nov 02 01:59:23.144657 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:23.144618    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3986fc99-87a5-41cf-bb18-0815f5e56d96-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm\" (UID: \"3986fc99-87a5-41cf-bb18-0815f5e56d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm"
Nov 02 01:59:23.145019 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:23.144994    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzv8x\" (UniqueName: \"kubernetes.io/projected/3986fc99-87a5-41cf-bb18-0815f5e56d96-kube-api-access-pzv8x\") pod \"maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm\" (UID: \"3986fc99-87a5-41cf-bb18-0815f5e56d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm"
Nov 02 01:59:23.145321 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:23.145301    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3986fc99-87a5-41cf-bb18-0815f5e56d96-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm\" (UID: \"3986fc99-87a5-41cf-bb18-0815f5e56d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm"
Nov 02 01:59:23.145491 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:23.145476    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3986fc99-87a5-41cf-bb18-0815f5e56d96-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm\" (UID: \"3986fc99-87a5-41cf-bb18-0815f5e56d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm"
Nov 02 01:59:23.260248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:23.255088    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3986fc99-87a5-41cf-bb18-0815f5e56d96-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm\" (UID: \"3986fc99-87a5-41cf-bb18-0815f5e56d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm"
Nov 02 01:59:23.260248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:23.255153    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzv8x\" (UniqueName: \"kubernetes.io/projected/3986fc99-87a5-41cf-bb18-0815f5e56d96-kube-api-access-pzv8x\") pod \"maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm\" (UID: \"3986fc99-87a5-41cf-bb18-0815f5e56d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm"
Nov 02 01:59:23.260248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:23.255265    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3986fc99-87a5-41cf-bb18-0815f5e56d96-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm\" (UID: \"3986fc99-87a5-41cf-bb18-0815f5e56d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm"
Nov 02 01:59:23.260248 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:23.255294    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3986fc99-87a5-41cf-bb18-0815f5e56d96-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm\" (UID: \"3986fc99-87a5-41cf-bb18-0815f5e56d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm"
Nov 02 01:59:23.412304 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:23.408712    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3986fc99-87a5-41cf-bb18-0815f5e56d96-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm\" (UID: \"3986fc99-87a5-41cf-bb18-0815f5e56d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm"
Nov 02 01:59:23.412561 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:59:23.412538    5875 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-serving-cert: secret "maas-api-e2e-shared-a-w5-73e1cc-serving-cert" not found
Nov 02 01:59:23.412743 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:59:23.412727    5875 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3986fc99-87a5-41cf-bb18-0815f5e56d96-maas-api-tls podName:3986fc99-87a5-41cf-bb18-0815f5e56d96 nodeName:}" failed. No retries permitted until 2026-11-02 01:59:23.912700117 +0000 UTC m=+1611.353736592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3986fc99-87a5-41cf-bb18-0815f5e56d96-maas-api-tls") pod "maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm" (UID: "3986fc99-87a5-41cf-bb18-0815f5e56d96") : secret "maas-api-e2e-shared-a-w5-73e1cc-serving-cert" not found
Nov 02 01:59:23.740370 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:23.739077    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3986fc99-87a5-41cf-bb18-0815f5e56d96-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm\" (UID: \"3986fc99-87a5-41cf-bb18-0815f5e56d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm"
Nov 02 01:59:23.741884 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:23.741555    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzv8x\" (UniqueName: \"kubernetes.io/projected/3986fc99-87a5-41cf-bb18-0815f5e56d96-kube-api-access-pzv8x\") pod \"maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm\" (UID: \"3986fc99-87a5-41cf-bb18-0815f5e56d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm"
Nov 02 01:59:24.015298 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:24.014446    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3986fc99-87a5-41cf-bb18-0815f5e56d96-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm\" (UID: \"3986fc99-87a5-41cf-bb18-0815f5e56d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm"
Nov 02 01:59:24.057247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:24.040841    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3986fc99-87a5-41cf-bb18-0815f5e56d96-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm\" (UID: \"3986fc99-87a5-41cf-bb18-0815f5e56d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm"
Nov 02 01:59:24.057247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:24.043493    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm"
Nov 02 01:59:24.720312 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:24.720251    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:59:24.758400 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:24.758356    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:59:25.243247 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:25.242972    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" podUID="6e39fc3d-d730-4df7-8d10-2376656a88d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:59:25.480472 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:25.464859    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm"]
Nov 02 01:59:26.075970 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:26.075845    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm" event={"ID":"3986fc99-87a5-41cf-bb18-0815f5e56d96","Type":"ContainerStarted","Data":"90b36da7466df29cc2ff1b21ccab247d27e082637e4d680d1398dab7d54978cf"}
Nov 02 01:59:26.159257 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:26.135349    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-lc78r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:59:26.159257 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:26.135424    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r" podUID="7cf2233f-a377-4bee-9eb9-5d5be2a7561c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:59:27.199319 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:27.195988    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm" event={"ID":"3986fc99-87a5-41cf-bb18-0815f5e56d96","Type":"ContainerStarted","Data":"19dd10fa0ddfbec1100b1affe90d4ccb0cb7a486dc125ab31e066a27aa4b304b"}
Nov 02 01:59:27.199319 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:27.198584    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm"
Nov 02 01:59:29.057786 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:29.057720    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm" podStartSLOduration=5.973873037 podStartE2EDuration="7.057699892s" totalImagesPullingTime="1.083826855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:59:27.349895744 +0000 UTC m=+1614.790932242" watchObservedRunningTime="2026-11-02 01:59:29.057699892 +0000 UTC m=+1616.498736402"
Nov 02 01:59:29.059938 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:29.059911    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39063960-xvqdf"]
Nov 02 01:59:29.116135 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:29.116094    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3dadcad-dqq47"]
Nov 02 01:59:29.120563 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:29.120523    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39063960-xvqdf"]
Nov 02 01:59:29.124320 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:29.124292    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3dadcad-dqq47"]
Nov 02 01:59:30.697833 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:30.697801    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4858d3ea-468d-447d-9f30-3cf259397bbb" path="/var/lib/kubelet/pods/4858d3ea-468d-447d-9f30-3cf259397bbb/volumes"
Nov 02 01:59:30.698484 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:30.698463    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddd3773f-67de-479a-97ff-666544bcb640" path="/var/lib/kubelet/pods/ddd3773f-67de-479a-97ff-666544bcb640/volumes"
Nov 02 01:59:33.253114 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:33.253086    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm"
Nov 02 01:59:33.302366 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:33.302329    5875 generic.go:356] "Generic (PLEG): container finished" podID="ff6cb777-acc6-4c58-8302-26d6e2906cdf" containerID="1af18b7499f5362c05ea1b91c9d3b741f1bf47f262cd701f6a6b9e10e0c3c86d" exitCode=0
Nov 02 01:59:33.302535 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:33.302404    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb" event={"ID":"ff6cb777-acc6-4c58-8302-26d6e2906cdf","Type":"ContainerDied","Data":"1af18b7499f5362c05ea1b91c9d3b741f1bf47f262cd701f6a6b9e10e0c3c86d"}
Nov 02 01:59:33.302913 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:33.302887    5875 scope.go:122] "RemoveContainer" containerID="1af18b7499f5362c05ea1b91c9d3b741f1bf47f262cd701f6a6b9e10e0c3c86d"
Nov 02 01:59:33.305139 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:33.305111    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:59:33.305282 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:33.305185    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:59:33.305282 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:33.305240    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:59:34.054130 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:34.054092    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7146e727-wzmhp"]
Nov 02 01:59:34.058013 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:34.057987    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7146e727-wzmhp"]
Nov 02 01:59:34.332257 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:34.332128    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb" event={"ID":"ff6cb777-acc6-4c58-8302-26d6e2906cdf","Type":"ContainerStarted","Data":"97de7d64ce55692ccda83e9d5866414b1d18cf9d5fc60159179055c4e5e2207a"}
Nov 02 01:59:34.683471 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:34.683388    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:59:34.684653 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:34.684627    5875 patch_prober.go:29] interesting pod/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 02 01:59:34.684784 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:34.684676    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb" podUID="ff6cb777-acc6-4c58-8302-26d6e2906cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 02 01:59:34.720420 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:34.720377    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46a20c98-086f-4f64-9a5a-5e2a2d0714df" path="/var/lib/kubelet/pods/46a20c98-086f-4f64-9a5a-5e2a2d0714df/volumes"
Nov 02 01:59:35.048056 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:35.048026    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07ab7724-l4cr4"]
Nov 02 01:59:35.051885 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:35.051863    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07ab7724-l4cr4"]
Nov 02 01:59:35.664972 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:35.664941    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:59:35.683230 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:35.683202    5875 patch_prober.go:29] interesting pod/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 02 01:59:35.683355 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:35.683291    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb" podUID="ff6cb777-acc6-4c58-8302-26d6e2906cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 02 01:59:36.683187 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:36.683157    5875 patch_prober.go:29] interesting pod/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 02 01:59:36.683615 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:36.683209    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb" podUID="ff6cb777-acc6-4c58-8302-26d6e2906cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 02 01:59:36.723245 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:36.719822    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0336b82f-b163-434d-952b-e37d9f8216a7" path="/var/lib/kubelet/pods/0336b82f-b163-434d-952b-e37d9f8216a7/volumes"
Nov 02 01:59:37.525703 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:37.525668    5875 scope.go:122] "RemoveContainer" containerID="de605ed6a0569ed9bff2823edf3f9259b0087945699e355df3e2a11e1f1d9a9a"
Nov 02 01:59:37.536705 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:37.536685    5875 scope.go:122] "RemoveContainer" containerID="eea042d275d86fa5ad00c353e506254a556dd9e36acc21aa9dc73ec16dd27b66"
Nov 02 01:59:37.546760 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:37.546738    5875 scope.go:122] "RemoveContainer" containerID="211f6f7ff706d60b71ee2f1cbb49c2b89096fa812b3ffca05594642acf0bab72"
Nov 02 01:59:37.683802 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:37.683768    5875 patch_prober.go:29] interesting pod/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 02 01:59:37.684187 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:37.683832    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb" podUID="ff6cb777-acc6-4c58-8302-26d6e2906cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 02 01:59:38.683432 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:38.683403    5875 patch_prober.go:29] interesting pod/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 02 01:59:38.683623 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:38.683477    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb" podUID="ff6cb777-acc6-4c58-8302-26d6e2906cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 02 01:59:39.683950 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:39.683916    5875 patch_prober.go:29] interesting pod/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 02 01:59:39.684350 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:39.683976    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb" podUID="ff6cb777-acc6-4c58-8302-26d6e2906cdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 02 01:59:40.697923 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:40.697845    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:59:40.699206 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:40.699183    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 01:59:41.319516 ip-10-0-141-68 kubenswrapper[5875]: E1102 01:59:41.319482    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 01:59:43.636751 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:43.636713    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-af807eae-tpnmf"]
Nov 02 01:59:43.642912 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:43.642889    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-af807eae-tpnmf"
Nov 02 01:59:43.700461 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:43.700427    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-af807eae-tpnmf"]
Nov 02 01:59:43.743671 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:43.743639    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 01:59:43.743890 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:43.743703    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 01:59:43.780682 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:43.780650    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f20d922-b401-4f5d-800b-64db3a04af32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-af807eae-tpnmf\" (UID: \"2f20d922-b401-4f5d-800b-64db3a04af32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-af807eae-tpnmf"
Nov 02 01:59:43.780921 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:43.780900    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj8q7\" (UniqueName: \"kubernetes.io/projected/2f20d922-b401-4f5d-800b-64db3a04af32-kube-api-access-zj8q7\") pod \"maas-api-revoke-keys-e2e-derive-af807eae-tpnmf\" (UID: \"2f20d922-b401-4f5d-800b-64db3a04af32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-af807eae-tpnmf"
Nov 02 01:59:43.780994 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:43.780944    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f20d922-b401-4f5d-800b-64db3a04af32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-af807eae-tpnmf\" (UID: \"2f20d922-b401-4f5d-800b-64db3a04af32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-af807eae-tpnmf"
Nov 02 01:59:43.882775 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:43.882743    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f20d922-b401-4f5d-800b-64db3a04af32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-af807eae-tpnmf\" (UID: \"2f20d922-b401-4f5d-800b-64db3a04af32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-af807eae-tpnmf"
Nov 02 01:59:43.882959 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:43.882909    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj8q7\" (UniqueName: \"kubernetes.io/projected/2f20d922-b401-4f5d-800b-64db3a04af32-kube-api-access-zj8q7\") pod \"maas-api-revoke-keys-e2e-derive-af807eae-tpnmf\" (UID: \"2f20d922-b401-4f5d-800b-64db3a04af32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-af807eae-tpnmf"
Nov 02 01:59:43.882959 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:43.882939    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f20d922-b401-4f5d-800b-64db3a04af32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-af807eae-tpnmf\" (UID: \"2f20d922-b401-4f5d-800b-64db3a04af32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-af807eae-tpnmf"
Nov 02 01:59:43.883669 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:43.883645    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f20d922-b401-4f5d-800b-64db3a04af32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-af807eae-tpnmf\" (UID: \"2f20d922-b401-4f5d-800b-64db3a04af32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-af807eae-tpnmf"
Nov 02 01:59:43.885265 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:43.885243    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f20d922-b401-4f5d-800b-64db3a04af32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-af807eae-tpnmf\" (UID: \"2f20d922-b401-4f5d-800b-64db3a04af32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-af807eae-tpnmf"
Nov 02 01:59:43.943805 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:43.943735    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj8q7\" (UniqueName: \"kubernetes.io/projected/2f20d922-b401-4f5d-800b-64db3a04af32-kube-api-access-zj8q7\") pod \"maas-api-revoke-keys-e2e-derive-af807eae-tpnmf\" (UID: \"2f20d922-b401-4f5d-800b-64db3a04af32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-af807eae-tpnmf"
Nov 02 01:59:43.957939 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:43.957915    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-af807eae-tpnmf"
Nov 02 01:59:44.344429 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:44.344242    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-af807eae-tpnmf"]
Nov 02 01:59:44.344580 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:59:44.344537    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f20d922_b401_4f5d_800b_64db3a04af32.slice/crio-6cca03d090c83b7ee3835b5ac3c162f1f704698803bc3fa694e32dac84619430 WatchSource:0}: Error finding container 6cca03d090c83b7ee3835b5ac3c162f1f704698803bc3fa694e32dac84619430: Status 404 returned error can't find the container with id 6cca03d090c83b7ee3835b5ac3c162f1f704698803bc3fa694e32dac84619430
Nov 02 01:59:44.489758 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:44.489720    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-af807eae-tpnmf" event={"ID":"2f20d922-b401-4f5d-800b-64db3a04af32","Type":"ContainerStarted","Data":"6cca03d090c83b7ee3835b5ac3c162f1f704698803bc3fa694e32dac84619430"}
Nov 02 01:59:45.048961 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:45.048877    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-761c34e8-7k6dh"]
Nov 02 01:59:45.097956 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:45.097825    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-761c34e8-7k6dh"]
Nov 02 01:59:45.501043 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:45.501005    5875 generic.go:356] "Generic (PLEG): container finished" podID="2f20d922-b401-4f5d-800b-64db3a04af32" containerID="670725bdf24dece7cfba264f997b88c7b5f696dc9b7b2e20ee4a2b1842fa61ec" exitCode=0
Nov 02 01:59:45.501326 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:45.501134    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-af807eae-tpnmf" event={"ID":"2f20d922-b401-4f5d-800b-64db3a04af32","Type":"ContainerDied","Data":"670725bdf24dece7cfba264f997b88c7b5f696dc9b7b2e20ee4a2b1842fa61ec"}
Nov 02 01:59:46.706897 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:46.706864    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0610139-0aec-49ea-a67f-e7815c2f6fe4" path="/var/lib/kubelet/pods/f0610139-0aec-49ea-a67f-e7815c2f6fe4/volumes"
Nov 02 01:59:46.793169 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:46.793093    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-af807eae-tpnmf"
Nov 02 01:59:46.828285 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:46.828255    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f20d922-b401-4f5d-800b-64db3a04af32-kube-api-access-zj8q7\" (UniqueName: \"kubernetes.io/projected/2f20d922-b401-4f5d-800b-64db3a04af32-kube-api-access-zj8q7\") pod \"2f20d922-b401-4f5d-800b-64db3a04af32\" (UID: \"2f20d922-b401-4f5d-800b-64db3a04af32\") "
Nov 02 01:59:46.828434 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:46.828337    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f20d922-b401-4f5d-800b-64db3a04af32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f20d922-b401-4f5d-800b-64db3a04af32-maas-api-xks-ca\") pod \"2f20d922-b401-4f5d-800b-64db3a04af32\" (UID: \"2f20d922-b401-4f5d-800b-64db3a04af32\") "
Nov 02 01:59:46.828475 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:46.828457    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f20d922-b401-4f5d-800b-64db3a04af32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f20d922-b401-4f5d-800b-64db3a04af32-maas-api-openshift-ca\") pod \"2f20d922-b401-4f5d-800b-64db3a04af32\" (UID: \"2f20d922-b401-4f5d-800b-64db3a04af32\") "
Nov 02 01:59:46.828821 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:46.828795    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f20d922-b401-4f5d-800b-64db3a04af32-maas-api-openshift-ca" pod "2f20d922-b401-4f5d-800b-64db3a04af32" (UID: "2f20d922-b401-4f5d-800b-64db3a04af32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:59:46.829145 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:46.829128    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f20d922-b401-4f5d-800b-64db3a04af32-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:46.830432 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:46.830409    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f20d922-b401-4f5d-800b-64db3a04af32-maas-api-xks-ca" pod "2f20d922-b401-4f5d-800b-64db3a04af32" (UID: "2f20d922-b401-4f5d-800b-64db3a04af32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:59:46.830534 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:46.830493    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f20d922-b401-4f5d-800b-64db3a04af32-kube-api-access-zj8q7" pod "2f20d922-b401-4f5d-800b-64db3a04af32" (UID: "2f20d922-b401-4f5d-800b-64db3a04af32"). InnerVolumeSpecName "kube-api-access-zj8q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:59:46.930931 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:46.930896    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f20d922-b401-4f5d-800b-64db3a04af32-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:46.930931 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:46.930933    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj8q7\" (UniqueName: \"kubernetes.io/projected/2f20d922-b401-4f5d-800b-64db3a04af32-kube-api-access-zj8q7\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:47.624262 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:47.618264    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-af807eae-tpnmf" event={"ID":"2f20d922-b401-4f5d-800b-64db3a04af32","Type":"ContainerDied","Data":"6cca03d090c83b7ee3835b5ac3c162f1f704698803bc3fa694e32dac84619430"}
Nov 02 01:59:47.624262 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:47.618335    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cca03d090c83b7ee3835b5ac3c162f1f704698803bc3fa694e32dac84619430"
Nov 02 01:59:47.624262 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:47.618403    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-af807eae-tpnmf"
Nov 02 01:59:49.638568 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:49.620330    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"]
Nov 02 01:59:49.640736 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:49.640709    5875 cpu_manager.go:414] "containerMap: removing container" podUID="2f20d922-b401-4f5d-800b-64db3a04af32" containerName="revoke-keys"
Nov 02 01:59:49.640931 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:49.640917    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f20d922-b401-4f5d-800b-64db3a04af32" containerName="revoke-keys"
Nov 02 01:59:49.641429 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:49.641412    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f20d922-b401-4f5d-800b-64db3a04af32" containerName="revoke-keys"
Nov 02 01:59:49.676609 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:49.676583    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:49.791254 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:49.761583    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5700d3d1-openshift-default-dockercfg-z5s4p\""
Nov 02 01:59:49.950315 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:49.829975    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c764b207-bd1e-4652-b4c1-519c8f453aa2-istiod-ca-cert\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.038955 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.038898    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-workload-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.039479 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.039454    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-envoy\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.039644 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.039629    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-podinfo\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.040454 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.040409    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-token\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.040785 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.040761    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb57p\" (UniqueName: \"kubernetes.io/projected/c764b207-bd1e-4652-b4c1-519c8f453aa2-kube-api-access-nb57p\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.042433 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.041351    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-workload-certs\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.042668 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.042648    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-data\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.042864 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.042842    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-credential-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.043308 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:49.901396    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"]
Nov 02 01:59:50.157612 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.153030    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb57p\" (UniqueName: \"kubernetes.io/projected/c764b207-bd1e-4652-b4c1-519c8f453aa2-kube-api-access-nb57p\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.157612 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.153134    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-workload-certs\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.157612 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.153215    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-data\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.157612 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.153310    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-credential-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.157612 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.153426    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c764b207-bd1e-4652-b4c1-519c8f453aa2-istiod-ca-cert\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.157612 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.153480    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-workload-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.157612 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.153518    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-envoy\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.157612 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.153550    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-podinfo\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.157612 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.153640    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-token\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.163011 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.162981    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c764b207-bd1e-4652-b4c1-519c8f453aa2-istiod-ca-cert\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.163734 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.163710    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-workload-certs\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.164139 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.164122    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-data\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.164541 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.164520    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-credential-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.185250 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.183166    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-workload-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.344161 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.344120    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb57p\" (UniqueName: \"kubernetes.io/projected/c764b207-bd1e-4652-b4c1-519c8f453aa2-kube-api-access-nb57p\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.362437 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.362407    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-podinfo\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.365459 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.365431    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-envoy\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.365647 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.365593    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-token\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:50.662397 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:50.662317    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:51.106569 ip-10-0-141-68 kubenswrapper[5875]: W1102 01:59:51.106526    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc764b207_bd1e_4652_b4c1_519c8f453aa2.slice/crio-72775ef79e14cbb905b79693bb3740dd134ab4fc395e1b2bb8128bf182353ec5 WatchSource:0}: Error finding container 72775ef79e14cbb905b79693bb3740dd134ab4fc395e1b2bb8128bf182353ec5: Status 404 returned error can't find the container with id 72775ef79e14cbb905b79693bb3740dd134ab4fc395e1b2bb8128bf182353ec5
Nov 02 01:59:51.107126 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:51.107101    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"]
Nov 02 01:59:51.109501 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:51.109467    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:59:51.109589 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:51.109533    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:59:51.109589 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:51.109566    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 01:59:51.705125 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:51.705094    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs" event={"ID":"c764b207-bd1e-4652-b4c1-519c8f453aa2","Type":"ContainerStarted","Data":"c06e8a7212c31d1e56abba5d488e65c1f67384f5d258f35e8f3dfa83c267712c"}
Nov 02 01:59:51.705546 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:51.705151    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs" event={"ID":"c764b207-bd1e-4652-b4c1-519c8f453aa2","Type":"ContainerStarted","Data":"72775ef79e14cbb905b79693bb3740dd134ab4fc395e1b2bb8128bf182353ec5"}
Nov 02 01:59:51.878201 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:51.878145    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs" podStartSLOduration=2.878130289 podStartE2EDuration="2.878130289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:59:51.874444058 +0000 UTC m=+1639.315480593" watchObservedRunningTime="2026-11-02 01:59:51.878130289 +0000 UTC m=+1639.319166786"
Nov 02 01:59:52.662670 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:52.662636    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:52.668816 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:52.668793    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:52.712905 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:52.712877    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:52.713707 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:52.713688    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 01:59:55.050962 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:55.050923    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1d5f2ad-mwkk7"]
Nov 02 01:59:55.100310 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:55.100283    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1d5f2ad-mwkk7"]
Nov 02 01:59:56.767063 ip-10-0-141-68 kubenswrapper[5875]: I1102 01:59:56.766945    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e532b09-28a0-464c-ad90-33d8aa7b4a0a" path="/var/lib/kubelet/pods/1e532b09-28a0-464c-ad90-33d8aa7b4a0a/volumes"
Nov 02 02:00:00.165403 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.165366    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f"]
Nov 02 02:00:00.173880 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.173852    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f"
Nov 02 02:00:00.177594 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.176460    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg"]
Nov 02 02:00:00.184082 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.184061    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874"]
Nov 02 02:00:00.184373 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.184353    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg"
Nov 02 02:00:00.190342 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.190323    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874"
Nov 02 02:00:00.233770 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.229292    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:00:00.260797 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.255465    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh"]
Nov 02 02:00:00.276376 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.274869    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbh5c\" (UniqueName: \"kubernetes.io/projected/f9e786f9-5769-42b9-8b85-90507b4fdc5a-kube-api-access-sbh5c\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg\" (UID: \"f9e786f9-5769-42b9-8b85-90507b4fdc5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg"
Nov 02 02:00:00.276376 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.274920    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdb61ec-f124-4f24-9ac4-94256e0ab23d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f\" (UID: \"3fdb61ec-f124-4f24-9ac4-94256e0ab23d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f"
Nov 02 02:00:00.276376 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.274937    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hggvz\" (UniqueName: \"kubernetes.io/projected/3fdb61ec-f124-4f24-9ac4-94256e0ab23d-kube-api-access-hggvz\") pod \"maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f\" (UID: \"3fdb61ec-f124-4f24-9ac4-94256e0ab23d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f"
Nov 02 02:00:00.276376 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.274972    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9e786f9-5769-42b9-8b85-90507b4fdc5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg\" (UID: \"f9e786f9-5769-42b9-8b85-90507b4fdc5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg"
Nov 02 02:00:00.276376 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.275081    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff0d1e0-810f-4091-92de-6e6575920d13-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893080-d7874\" (UID: \"8ff0d1e0-810f-4091-92de-6e6575920d13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874"
Nov 02 02:00:00.276376 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.275156    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4qqv\" (UniqueName: \"kubernetes.io/projected/8ff0d1e0-810f-4091-92de-6e6575920d13-kube-api-access-x4qqv\") pod \"maas-api-key-cleanup-29893080-d7874\" (UID: \"8ff0d1e0-810f-4091-92de-6e6575920d13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874"
Nov 02 02:00:00.277440 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.277415    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k"]
Nov 02 02:00:00.277630 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.277606    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh"
Nov 02 02:00:00.283646 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.283626    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f"]
Nov 02 02:00:00.284176 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.283857    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k"
Nov 02 02:00:00.287607 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.287585    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg"]
Nov 02 02:00:00.379166 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.379136    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbh5c\" (UniqueName: \"kubernetes.io/projected/f9e786f9-5769-42b9-8b85-90507b4fdc5a-kube-api-access-sbh5c\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg\" (UID: \"f9e786f9-5769-42b9-8b85-90507b4fdc5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg"
Nov 02 02:00:00.379608 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.379578    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdb61ec-f124-4f24-9ac4-94256e0ab23d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f\" (UID: \"3fdb61ec-f124-4f24-9ac4-94256e0ab23d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f"
Nov 02 02:00:00.379746 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.379729    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hggvz\" (UniqueName: \"kubernetes.io/projected/3fdb61ec-f124-4f24-9ac4-94256e0ab23d-kube-api-access-hggvz\") pod \"maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f\" (UID: \"3fdb61ec-f124-4f24-9ac4-94256e0ab23d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f"
Nov 02 02:00:00.379860 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.379848    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9e786f9-5769-42b9-8b85-90507b4fdc5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg\" (UID: \"f9e786f9-5769-42b9-8b85-90507b4fdc5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg"
Nov 02 02:00:00.379997 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.379983    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/025b1e27-ad55-46ed-a362-ab9e23c4d6d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh\" (UID: \"025b1e27-ad55-46ed-a362-ab9e23c4d6d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh"
Nov 02 02:00:00.380112 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.380100    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff0d1e0-810f-4091-92de-6e6575920d13-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893080-d7874\" (UID: \"8ff0d1e0-810f-4091-92de-6e6575920d13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874"
Nov 02 02:00:00.380257 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.380242    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4qqv\" (UniqueName: \"kubernetes.io/projected/8ff0d1e0-810f-4091-92de-6e6575920d13-kube-api-access-x4qqv\") pod \"maas-api-key-cleanup-29893080-d7874\" (UID: \"8ff0d1e0-810f-4091-92de-6e6575920d13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874"
Nov 02 02:00:00.380368 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.380355    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/927e4a13-38c2-4b2c-a630-35db116a5ed3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k\" (UID: \"927e4a13-38c2-4b2c-a630-35db116a5ed3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k"
Nov 02 02:00:00.380472 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.380460    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj7fl\" (UniqueName: \"kubernetes.io/projected/927e4a13-38c2-4b2c-a630-35db116a5ed3-kube-api-access-wj7fl\") pod \"maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k\" (UID: \"927e4a13-38c2-4b2c-a630-35db116a5ed3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k"
Nov 02 02:00:00.380557 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.380532    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdb61ec-f124-4f24-9ac4-94256e0ab23d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f\" (UID: \"3fdb61ec-f124-4f24-9ac4-94256e0ab23d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f"
Nov 02 02:00:00.380776 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.380762    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92whj\" (UniqueName: \"kubernetes.io/projected/025b1e27-ad55-46ed-a362-ab9e23c4d6d5-kube-api-access-92whj\") pod \"maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh\" (UID: \"025b1e27-ad55-46ed-a362-ab9e23c4d6d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh"
Nov 02 02:00:00.381936 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.381916    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9e786f9-5769-42b9-8b85-90507b4fdc5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg\" (UID: \"f9e786f9-5769-42b9-8b85-90507b4fdc5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg"
Nov 02 02:00:00.386483 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.386456    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff0d1e0-810f-4091-92de-6e6575920d13-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893080-d7874\" (UID: \"8ff0d1e0-810f-4091-92de-6e6575920d13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874"
Nov 02 02:00:00.393525 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.393496    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874"]
Nov 02 02:00:00.462167 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.462131    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4qqv\" (UniqueName: \"kubernetes.io/projected/8ff0d1e0-810f-4091-92de-6e6575920d13-kube-api-access-x4qqv\") pod \"maas-api-key-cleanup-29893080-d7874\" (UID: \"8ff0d1e0-810f-4091-92de-6e6575920d13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874"
Nov 02 02:00:00.464173 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.464101    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbh5c\" (UniqueName: \"kubernetes.io/projected/f9e786f9-5769-42b9-8b85-90507b4fdc5a-kube-api-access-sbh5c\") pod \"maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg\" (UID: \"f9e786f9-5769-42b9-8b85-90507b4fdc5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg"
Nov 02 02:00:00.466353 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.466330    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh"]
Nov 02 02:00:00.467320 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.467274    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hggvz\" (UniqueName: \"kubernetes.io/projected/3fdb61ec-f124-4f24-9ac4-94256e0ab23d-kube-api-access-hggvz\") pod \"maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f\" (UID: \"3fdb61ec-f124-4f24-9ac4-94256e0ab23d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f"
Nov 02 02:00:00.474057 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.474038    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k"]
Nov 02 02:00:00.482589 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.482561    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/025b1e27-ad55-46ed-a362-ab9e23c4d6d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh\" (UID: \"025b1e27-ad55-46ed-a362-ab9e23c4d6d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh"
Nov 02 02:00:00.482722 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.482652    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/927e4a13-38c2-4b2c-a630-35db116a5ed3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k\" (UID: \"927e4a13-38c2-4b2c-a630-35db116a5ed3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k"
Nov 02 02:00:00.482722 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.482693    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj7fl\" (UniqueName: \"kubernetes.io/projected/927e4a13-38c2-4b2c-a630-35db116a5ed3-kube-api-access-wj7fl\") pod \"maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k\" (UID: \"927e4a13-38c2-4b2c-a630-35db116a5ed3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k"
Nov 02 02:00:00.483387 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.482810    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92whj\" (UniqueName: \"kubernetes.io/projected/025b1e27-ad55-46ed-a362-ab9e23c4d6d5-kube-api-access-92whj\") pod \"maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh\" (UID: \"025b1e27-ad55-46ed-a362-ab9e23c4d6d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh"
Nov 02 02:00:00.483669 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.483570    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/927e4a13-38c2-4b2c-a630-35db116a5ed3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k\" (UID: \"927e4a13-38c2-4b2c-a630-35db116a5ed3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k"
Nov 02 02:00:00.483669 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.483570    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/025b1e27-ad55-46ed-a362-ab9e23c4d6d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh\" (UID: \"025b1e27-ad55-46ed-a362-ab9e23c4d6d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh"
Nov 02 02:00:00.557569 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.554612    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92whj\" (UniqueName: \"kubernetes.io/projected/025b1e27-ad55-46ed-a362-ab9e23c4d6d5-kube-api-access-92whj\") pod \"maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh\" (UID: \"025b1e27-ad55-46ed-a362-ab9e23c4d6d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh"
Nov 02 02:00:00.561452 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.560885    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj7fl\" (UniqueName: \"kubernetes.io/projected/927e4a13-38c2-4b2c-a630-35db116a5ed3-kube-api-access-wj7fl\") pod \"maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k\" (UID: \"927e4a13-38c2-4b2c-a630-35db116a5ed3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k"
Nov 02 02:00:00.561452 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.561182    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f"
Nov 02 02:00:00.577791 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.575838    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg"
Nov 02 02:00:00.585474 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.585444    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874"
Nov 02 02:00:00.677994 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.677969    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh"
Nov 02 02:00:00.760821 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:00.760786    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k"
Nov 02 02:00:01.289344 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:01.289313    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f"]
Nov 02 02:00:01.358431 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:01.358349    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg"]
Nov 02 02:00:01.370898 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:00:01.370864    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e786f9_5769_42b9_8b85_90507b4fdc5a.slice/crio-cc64e91e84b33a659c87c6fdbfec41aefcd8379c4359b56ebd3f4ba9de9a09f7 WatchSource:0}: Error finding container cc64e91e84b33a659c87c6fdbfec41aefcd8379c4359b56ebd3f4ba9de9a09f7: Status 404 returned error can't find the container with id cc64e91e84b33a659c87c6fdbfec41aefcd8379c4359b56ebd3f4ba9de9a09f7
Nov 02 02:00:01.395948 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:00:01.395915    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff0d1e0_810f_4091_92de_6e6575920d13.slice/crio-7779e49e4db2fadd16a375f976eea0db2e7db73710a43e01198345995e9e2931 WatchSource:0}: Error finding container 7779e49e4db2fadd16a375f976eea0db2e7db73710a43e01198345995e9e2931: Status 404 returned error can't find the container with id 7779e49e4db2fadd16a375f976eea0db2e7db73710a43e01198345995e9e2931
Nov 02 02:00:01.396602 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:01.396569    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874"]
Nov 02 02:00:01.556251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:01.513406    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k"]
Nov 02 02:00:01.560122 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:01.560082    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh"]
Nov 02 02:00:01.977997 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:01.977899    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874" event={"ID":"8ff0d1e0-810f-4091-92de-6e6575920d13","Type":"ContainerStarted","Data":"7779e49e4db2fadd16a375f976eea0db2e7db73710a43e01198345995e9e2931"}
Nov 02 02:00:01.979416 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:01.979388    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh" event={"ID":"025b1e27-ad55-46ed-a362-ab9e23c4d6d5","Type":"ContainerStarted","Data":"f7e5cb01d75579d89158f508d8e9c99b4b23f5161916ca9125e9fc8a9853ca0b"}
Nov 02 02:00:01.981046 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:01.981019    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f" event={"ID":"3fdb61ec-f124-4f24-9ac4-94256e0ab23d","Type":"ContainerStarted","Data":"ca8d7e9fc62a593fb61644445fd05ae7b706704ffa133efe665a114ce18c8ec9"}
Nov 02 02:00:01.981155 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:01.981069    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f" event={"ID":"3fdb61ec-f124-4f24-9ac4-94256e0ab23d","Type":"ContainerStarted","Data":"815aae56282273a73dda60c956bda186e668689b8b85a1706fb87964d5c1e629"}
Nov 02 02:00:01.982511 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:01.982484    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg" event={"ID":"f9e786f9-5769-42b9-8b85-90507b4fdc5a","Type":"ContainerStarted","Data":"cc64e91e84b33a659c87c6fdbfec41aefcd8379c4359b56ebd3f4ba9de9a09f7"}
Nov 02 02:00:01.986406 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:01.986371    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k" event={"ID":"927e4a13-38c2-4b2c-a630-35db116a5ed3","Type":"ContainerStarted","Data":"554b390609f614ba3122e7499d9b24cba2c54b9ebfcc58254caec4d67f0445ac"}
Nov 02 02:00:02.127247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:02.123596    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f" podStartSLOduration=2.123573516 podStartE2EDuration="2.123573516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:02.10962737 +0000 UTC m=+1649.550663870" watchObservedRunningTime="2026-11-02 02:00:02.123573516 +0000 UTC m=+1649.564610014"
Nov 02 02:00:03.017584 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:03.017545    5875 generic.go:356] "Generic (PLEG): container finished" podID="8ff0d1e0-810f-4091-92de-6e6575920d13" containerID="2d7987721348ec3ba11a5911f92958eb3867c722e38f0a9e74ec6bb065cabb82" exitCode=60
Nov 02 02:00:03.018205 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:03.018178    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874" event={"ID":"8ff0d1e0-810f-4091-92de-6e6575920d13","Type":"ContainerDied","Data":"2d7987721348ec3ba11a5911f92958eb3867c722e38f0a9e74ec6bb065cabb82"}
Nov 02 02:00:03.018841 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:03.018822    5875 scope.go:122] "RemoveContainer" containerID="2d7987721348ec3ba11a5911f92958eb3867c722e38f0a9e74ec6bb065cabb82"
Nov 02 02:00:03.048315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:03.048283    5875 generic.go:356] "Generic (PLEG): container finished" podID="025b1e27-ad55-46ed-a362-ab9e23c4d6d5" containerID="0d76a892e48114d67661972da95311fb4b1bb98d1536f37211cfbe219e88d930" exitCode=60
Nov 02 02:00:03.048803 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:03.048544    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh" event={"ID":"025b1e27-ad55-46ed-a362-ab9e23c4d6d5","Type":"ContainerDied","Data":"0d76a892e48114d67661972da95311fb4b1bb98d1536f37211cfbe219e88d930"}
Nov 02 02:00:03.052622 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:03.052130    5875 scope.go:122] "RemoveContainer" containerID="0d76a892e48114d67661972da95311fb4b1bb98d1536f37211cfbe219e88d930"
Nov 02 02:00:03.057701 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:03.056895    5875 generic.go:356] "Generic (PLEG): container finished" podID="3fdb61ec-f124-4f24-9ac4-94256e0ab23d" containerID="ca8d7e9fc62a593fb61644445fd05ae7b706704ffa133efe665a114ce18c8ec9" exitCode=60
Nov 02 02:00:03.057701 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:03.057103    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f" event={"ID":"3fdb61ec-f124-4f24-9ac4-94256e0ab23d","Type":"ContainerDied","Data":"ca8d7e9fc62a593fb61644445fd05ae7b706704ffa133efe665a114ce18c8ec9"}
Nov 02 02:00:03.057701 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:03.057496    5875 scope.go:122] "RemoveContainer" containerID="ca8d7e9fc62a593fb61644445fd05ae7b706704ffa133efe665a114ce18c8ec9"
Nov 02 02:00:03.059948 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:03.059922    5875 generic.go:356] "Generic (PLEG): container finished" podID="f9e786f9-5769-42b9-8b85-90507b4fdc5a" containerID="8256d0e9bad6a5744d36c222d8556c46e9a6f9a46a0cf68327c8c583a6d1f32a" exitCode=60
Nov 02 02:00:03.060058 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:03.060034    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg" event={"ID":"f9e786f9-5769-42b9-8b85-90507b4fdc5a","Type":"ContainerDied","Data":"8256d0e9bad6a5744d36c222d8556c46e9a6f9a46a0cf68327c8c583a6d1f32a"}
Nov 02 02:00:03.060278 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:03.060260    5875 scope.go:122] "RemoveContainer" containerID="8256d0e9bad6a5744d36c222d8556c46e9a6f9a46a0cf68327c8c583a6d1f32a"
Nov 02 02:00:03.062257 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:03.062006    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k" event={"ID":"927e4a13-38c2-4b2c-a630-35db116a5ed3","Type":"ContainerStarted","Data":"24abcf415016026b504d227d3889745b00171c06273c69b6ea5a568af4a4e8cb"}
Nov 02 02:00:03.657781 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:03.656078    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k" podStartSLOduration=3.6560586969999997 podStartE2EDuration="3.656058697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:03.655676813 +0000 UTC m=+1651.096713313" watchObservedRunningTime="2026-11-02 02:00:03.656058697 +0000 UTC m=+1651.097095194"
Nov 02 02:00:04.073072 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:04.073028    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh" event={"ID":"025b1e27-ad55-46ed-a362-ab9e23c4d6d5","Type":"ContainerStarted","Data":"2e541361724db20ea201e0d027ee1490aef0234591d6eb28ef5e264dc36f312a"}
Nov 02 02:00:04.075155 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:04.075127    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f" event={"ID":"3fdb61ec-f124-4f24-9ac4-94256e0ab23d","Type":"ContainerStarted","Data":"08000c17a58300270b8a46e57944a1d39488b7ab7a977afa5602eaac118e03fd"}
Nov 02 02:00:04.077160 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:04.077139    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg" event={"ID":"f9e786f9-5769-42b9-8b85-90507b4fdc5a","Type":"ContainerStarted","Data":"a8e2e1f3accb45b044ba42eb84b2e80345921d54c34d5e9f40f3bb1cdc5b4dd1"}
Nov 02 02:00:04.079267 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:04.079208    5875 generic.go:356] "Generic (PLEG): container finished" podID="927e4a13-38c2-4b2c-a630-35db116a5ed3" containerID="24abcf415016026b504d227d3889745b00171c06273c69b6ea5a568af4a4e8cb" exitCode=60
Nov 02 02:00:04.079346 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:04.079264    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k" event={"ID":"927e4a13-38c2-4b2c-a630-35db116a5ed3","Type":"ContainerDied","Data":"24abcf415016026b504d227d3889745b00171c06273c69b6ea5a568af4a4e8cb"}
Nov 02 02:00:04.079560 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:04.079542    5875 scope.go:122] "RemoveContainer" containerID="24abcf415016026b504d227d3889745b00171c06273c69b6ea5a568af4a4e8cb"
Nov 02 02:00:04.081985 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:04.081947    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874" event={"ID":"8ff0d1e0-810f-4091-92de-6e6575920d13","Type":"ContainerStarted","Data":"bfef1fffb9350441a209f78043052804a6bc89456e4cf487e61274fd43f9e747"}
Nov 02 02:00:04.370897 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:04.370568    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh" podStartSLOduration=4.370554028 podStartE2EDuration="4.370554028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:04.246345972 +0000 UTC m=+1651.687382469" watchObservedRunningTime="2026-11-02 02:00:04.370554028 +0000 UTC m=+1651.811590524"
Nov 02 02:00:04.371648 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:04.371612    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874" podStartSLOduration=4.371603299 podStartE2EDuration="4.371603299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:04.368658395 +0000 UTC m=+1651.809694888" watchObservedRunningTime="2026-11-02 02:00:04.371603299 +0000 UTC m=+1651.812639832"
Nov 02 02:00:04.647715 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:04.647653    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg" podStartSLOduration=4.647633053 podStartE2EDuration="4.647633053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:04.646250516 +0000 UTC m=+1652.087286994" watchObservedRunningTime="2026-11-02 02:00:04.647633053 +0000 UTC m=+1652.088669551"
Nov 02 02:00:05.094437 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:05.094406    5875 generic.go:356] "Generic (PLEG): container finished" podID="8ff0d1e0-810f-4091-92de-6e6575920d13" containerID="bfef1fffb9350441a209f78043052804a6bc89456e4cf487e61274fd43f9e747" exitCode=60
Nov 02 02:00:05.095061 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:05.094491    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874" event={"ID":"8ff0d1e0-810f-4091-92de-6e6575920d13","Type":"ContainerDied","Data":"bfef1fffb9350441a209f78043052804a6bc89456e4cf487e61274fd43f9e747"}
Nov 02 02:00:05.095061 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:05.094535    5875 scope.go:122] "RemoveContainer" containerID="2d7987721348ec3ba11a5911f92958eb3867c722e38f0a9e74ec6bb065cabb82"
Nov 02 02:00:05.095061 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:05.094874    5875 scope.go:122] "RemoveContainer" containerID="bfef1fffb9350441a209f78043052804a6bc89456e4cf487e61274fd43f9e747"
Nov 02 02:00:05.095199 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:00:05.095157    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893080-d7874_odh-ai-gateway-infra(8ff0d1e0-810f-4091-92de-6e6575920d13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874" podUID="8ff0d1e0-810f-4091-92de-6e6575920d13"
Nov 02 02:00:05.118267 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:05.117943    5875 generic.go:356] "Generic (PLEG): container finished" podID="025b1e27-ad55-46ed-a362-ab9e23c4d6d5" containerID="2e541361724db20ea201e0d027ee1490aef0234591d6eb28ef5e264dc36f312a" exitCode=60
Nov 02 02:00:05.118267 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:05.118070    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh" event={"ID":"025b1e27-ad55-46ed-a362-ab9e23c4d6d5","Type":"ContainerDied","Data":"2e541361724db20ea201e0d027ee1490aef0234591d6eb28ef5e264dc36f312a"}
Nov 02 02:00:05.123256 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:05.118605    5875 scope.go:122] "RemoveContainer" containerID="2e541361724db20ea201e0d027ee1490aef0234591d6eb28ef5e264dc36f312a"
Nov 02 02:00:05.123256 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:00:05.119070    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh_odh-ai-gateway-infra(025b1e27-ad55-46ed-a362-ab9e23c4d6d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh" podUID="025b1e27-ad55-46ed-a362-ab9e23c4d6d5"
Nov 02 02:00:05.149848 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:05.149783    5875 generic.go:356] "Generic (PLEG): container finished" podID="3fdb61ec-f124-4f24-9ac4-94256e0ab23d" containerID="08000c17a58300270b8a46e57944a1d39488b7ab7a977afa5602eaac118e03fd" exitCode=60
Nov 02 02:00:05.150045 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:05.150024    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f" event={"ID":"3fdb61ec-f124-4f24-9ac4-94256e0ab23d","Type":"ContainerDied","Data":"08000c17a58300270b8a46e57944a1d39488b7ab7a977afa5602eaac118e03fd"}
Nov 02 02:00:05.152255 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:05.150556    5875 scope.go:122] "RemoveContainer" containerID="08000c17a58300270b8a46e57944a1d39488b7ab7a977afa5602eaac118e03fd"
Nov 02 02:00:05.152255 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:00:05.151031    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f_odh-ai-gateway-infra(3fdb61ec-f124-4f24-9ac4-94256e0ab23d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f" podUID="3fdb61ec-f124-4f24-9ac4-94256e0ab23d"
Nov 02 02:00:05.159262 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:05.156643    5875 generic.go:356] "Generic (PLEG): container finished" podID="f9e786f9-5769-42b9-8b85-90507b4fdc5a" containerID="a8e2e1f3accb45b044ba42eb84b2e80345921d54c34d5e9f40f3bb1cdc5b4dd1" exitCode=60
Nov 02 02:00:05.159262 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:05.156753    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg" event={"ID":"f9e786f9-5769-42b9-8b85-90507b4fdc5a","Type":"ContainerDied","Data":"a8e2e1f3accb45b044ba42eb84b2e80345921d54c34d5e9f40f3bb1cdc5b4dd1"}
Nov 02 02:00:05.159262 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:05.157187    5875 scope.go:122] "RemoveContainer" containerID="a8e2e1f3accb45b044ba42eb84b2e80345921d54c34d5e9f40f3bb1cdc5b4dd1"
Nov 02 02:00:05.159262 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:00:05.157615    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg_odh-ai-gateway-infra(f9e786f9-5769-42b9-8b85-90507b4fdc5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg" podUID="f9e786f9-5769-42b9-8b85-90507b4fdc5a"
Nov 02 02:00:05.160994 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:05.160120    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k" event={"ID":"927e4a13-38c2-4b2c-a630-35db116a5ed3","Type":"ContainerStarted","Data":"5d92e21f69fd4c2a8890928cc7e1a1d3578c83c412161b70c147742dada55695"}
Nov 02 02:00:05.170035 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:05.170009    5875 scope.go:122] "RemoveContainer" containerID="0d76a892e48114d67661972da95311fb4b1bb98d1536f37211cfbe219e88d930"
Nov 02 02:00:05.186140 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:05.186123    5875 scope.go:122] "RemoveContainer" containerID="ca8d7e9fc62a593fb61644445fd05ae7b706704ffa133efe665a114ce18c8ec9"
Nov 02 02:00:05.233799 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:05.233776    5875 scope.go:122] "RemoveContainer" containerID="8256d0e9bad6a5744d36c222d8556c46e9a6f9a46a0cf68327c8c583a6d1f32a"
Nov 02 02:00:06.175746 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:06.175710    5875 scope.go:122] "RemoveContainer" containerID="bfef1fffb9350441a209f78043052804a6bc89456e4cf487e61274fd43f9e747"
Nov 02 02:00:06.176261 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:00:06.176177    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893080-d7874_odh-ai-gateway-infra(8ff0d1e0-810f-4091-92de-6e6575920d13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874" podUID="8ff0d1e0-810f-4091-92de-6e6575920d13"
Nov 02 02:00:06.177588 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:06.177563    5875 scope.go:122] "RemoveContainer" containerID="2e541361724db20ea201e0d027ee1490aef0234591d6eb28ef5e264dc36f312a"
Nov 02 02:00:06.177998 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:00:06.177968    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh_odh-ai-gateway-infra(025b1e27-ad55-46ed-a362-ab9e23c4d6d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh" podUID="025b1e27-ad55-46ed-a362-ab9e23c4d6d5"
Nov 02 02:00:06.179374 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:06.179353    5875 scope.go:122] "RemoveContainer" containerID="08000c17a58300270b8a46e57944a1d39488b7ab7a977afa5602eaac118e03fd"
Nov 02 02:00:06.179739 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:00:06.179716    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f_odh-ai-gateway-infra(3fdb61ec-f124-4f24-9ac4-94256e0ab23d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f" podUID="3fdb61ec-f124-4f24-9ac4-94256e0ab23d"
Nov 02 02:00:06.181200 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:06.181181    5875 scope.go:122] "RemoveContainer" containerID="a8e2e1f3accb45b044ba42eb84b2e80345921d54c34d5e9f40f3bb1cdc5b4dd1"
Nov 02 02:00:06.181546 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:00:06.181520    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg_odh-ai-gateway-infra(f9e786f9-5769-42b9-8b85-90507b4fdc5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg" podUID="f9e786f9-5769-42b9-8b85-90507b4fdc5a"
Nov 02 02:00:06.182957 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:06.182928    5875 generic.go:356] "Generic (PLEG): container finished" podID="927e4a13-38c2-4b2c-a630-35db116a5ed3" containerID="5d92e21f69fd4c2a8890928cc7e1a1d3578c83c412161b70c147742dada55695" exitCode=60
Nov 02 02:00:06.183075 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:06.182968    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k" event={"ID":"927e4a13-38c2-4b2c-a630-35db116a5ed3","Type":"ContainerDied","Data":"5d92e21f69fd4c2a8890928cc7e1a1d3578c83c412161b70c147742dada55695"}
Nov 02 02:00:06.183075 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:06.183012    5875 scope.go:122] "RemoveContainer" containerID="24abcf415016026b504d227d3889745b00171c06273c69b6ea5a568af4a4e8cb"
Nov 02 02:00:06.183284 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:06.183264    5875 scope.go:122] "RemoveContainer" containerID="5d92e21f69fd4c2a8890928cc7e1a1d3578c83c412161b70c147742dada55695"
Nov 02 02:00:06.183530 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:00:06.183512    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k_odh-ai-gateway-infra(927e4a13-38c2-4b2c-a630-35db116a5ed3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k" podUID="927e4a13-38c2-4b2c-a630-35db116a5ed3"
Nov 02 02:00:07.196908 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:07.196879    5875 scope.go:122] "RemoveContainer" containerID="5d92e21f69fd4c2a8890928cc7e1a1d3578c83c412161b70c147742dada55695"
Nov 02 02:00:07.197390 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:00:07.197366    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k_odh-ai-gateway-infra(927e4a13-38c2-4b2c-a630-35db116a5ed3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k" podUID="927e4a13-38c2-4b2c-a630-35db116a5ed3"
Nov 02 02:00:08.874431 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:08.874391    5875 patch_prober.go:29] interesting pod/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:00:08.874906 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:08.874463    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f" podUID="eedec700-8816-4b96-a087-7d0d691123bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:00:11.320388 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:00:11.320339    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:00:12.368376 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:12.368341    5875 generic.go:356] "Generic (PLEG): container finished" podID="eedec700-8816-4b96-a087-7d0d691123bf" containerID="7473807fe018380eaa349742a6a3139a3dd4262b48a624125925debf92ce53b4" exitCode=0
Nov 02 02:00:12.368851 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:12.368417    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f" event={"ID":"eedec700-8816-4b96-a087-7d0d691123bf","Type":"ContainerDied","Data":"7473807fe018380eaa349742a6a3139a3dd4262b48a624125925debf92ce53b4"}
Nov 02 02:00:12.368929 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:12.368900    5875 scope.go:122] "RemoveContainer" containerID="7473807fe018380eaa349742a6a3139a3dd4262b48a624125925debf92ce53b4"
Nov 02 02:00:12.370375 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:12.370340    5875 generic.go:356] "Generic (PLEG): container finished" podID="d4b7a180-01b7-4c32-93be-dcb96b6b3b42" containerID="639b38cd3662735e99dce69faf3e49b82f5b265b24507010e1ef42dc813016a9" exitCode=0
Nov 02 02:00:12.370494 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:12.370368    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf" event={"ID":"d4b7a180-01b7-4c32-93be-dcb96b6b3b42","Type":"ContainerDied","Data":"639b38cd3662735e99dce69faf3e49b82f5b265b24507010e1ef42dc813016a9"}
Nov 02 02:00:12.370758 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:12.370728    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:00:12.370826 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:12.370811    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:00:12.370869 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:12.370748    5875 scope.go:122] "RemoveContainer" containerID="639b38cd3662735e99dce69faf3e49b82f5b265b24507010e1ef42dc813016a9"
Nov 02 02:00:12.370869 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:12.370856    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:00:12.372406 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:12.372382    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:00:12.372502 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:12.372439    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:00:12.372502 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:12.372478    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:00:13.381182 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:13.381145    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f" event={"ID":"eedec700-8816-4b96-a087-7d0d691123bf","Type":"ContainerStarted","Data":"e2c1c23aa5ab53e5abc5f336a306437262145fecf70ce32c55303cdfc05a87ee"}
Nov 02 02:00:13.383307 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:13.383284    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf" event={"ID":"d4b7a180-01b7-4c32-93be-dcb96b6b3b42","Type":"ContainerStarted","Data":"f0a92bdc72fd824df97fc3528a7b6276d955dc32e5d8f2db4f91f239bdb8ba46"}
Nov 02 02:00:13.873369 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:13.873334    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 02:00:13.874567 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:13.874545    5875 patch_prober.go:29] interesting pod/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Nov 02 02:00:13.874663 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:13.874602    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf" podUID="d4b7a180-01b7-4c32-93be-dcb96b6b3b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Nov 02 02:00:14.289270 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:14.289209    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 02:00:14.289477 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:14.289453    5875 patch_prober.go:29] interesting pod/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 02 02:00:14.289542 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:14.289520    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f" podUID="eedec700-8816-4b96-a087-7d0d691123bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 02 02:00:14.347673 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:14.347573    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 02:00:14.872778 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:14.872744    5875 patch_prober.go:29] interesting pod/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Nov 02 02:00:14.873167 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:14.872801    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf" podUID="d4b7a180-01b7-4c32-93be-dcb96b6b3b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Nov 02 02:00:15.288916 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:15.288879    5875 patch_prober.go:29] interesting pod/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 02 02:00:15.289098 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:15.288935    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f" podUID="eedec700-8816-4b96-a087-7d0d691123bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 02 02:00:15.687343 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:15.687262    5875 scope.go:122] "RemoveContainer" containerID="a8e2e1f3accb45b044ba42eb84b2e80345921d54c34d5e9f40f3bb1cdc5b4dd1"
Nov 02 02:00:15.687343 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:15.687326    5875 scope.go:122] "RemoveContainer" containerID="bfef1fffb9350441a209f78043052804a6bc89456e4cf487e61274fd43f9e747"
Nov 02 02:00:15.873193 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:15.873164    5875 patch_prober.go:29] interesting pod/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Nov 02 02:00:15.873631 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:15.873216    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf" podUID="d4b7a180-01b7-4c32-93be-dcb96b6b3b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Nov 02 02:00:16.289253 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:16.289151    5875 patch_prober.go:29] interesting pod/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 02 02:00:16.289253 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:16.289212    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f" podUID="eedec700-8816-4b96-a087-7d0d691123bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 02 02:00:16.467072 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:16.467030    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg" event={"ID":"f9e786f9-5769-42b9-8b85-90507b4fdc5a","Type":"ContainerStarted","Data":"c3b7b388fd542eb0ee42a7e81c08da50992032235c8b6cce59c7b9f5c7728317"}
Nov 02 02:00:16.471125 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:16.471097    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874" event={"ID":"8ff0d1e0-810f-4091-92de-6e6575920d13","Type":"ContainerStarted","Data":"bb6ac0773cc6e2cedbe5fcc49b75d2c7058305ab6f8ad1b8e614a3696713da1b"}
Nov 02 02:00:16.872961 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:16.872913    5875 patch_prober.go:29] interesting pod/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Nov 02 02:00:16.873188 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:16.872986    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf" podUID="d4b7a180-01b7-4c32-93be-dcb96b6b3b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Nov 02 02:00:17.288834 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:17.288801    5875 patch_prober.go:29] interesting pod/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 02 02:00:17.289279 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:17.288862    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f" podUID="eedec700-8816-4b96-a087-7d0d691123bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 02 02:00:17.743248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:17.737886    5875 scope.go:122] "RemoveContainer" containerID="08000c17a58300270b8a46e57944a1d39488b7ab7a977afa5602eaac118e03fd"
Nov 02 02:00:17.766246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:17.765567    5875 scope.go:122] "RemoveContainer" containerID="5d92e21f69fd4c2a8890928cc7e1a1d3578c83c412161b70c147742dada55695"
Nov 02 02:00:17.783479 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:17.783456    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg"]
Nov 02 02:00:17.787183 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:17.787157    5875 generic.go:356] "Generic (PLEG): container finished" podID="f9e786f9-5769-42b9-8b85-90507b4fdc5a" containerID="c3b7b388fd542eb0ee42a7e81c08da50992032235c8b6cce59c7b9f5c7728317" exitCode=60
Nov 02 02:00:17.787356 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:17.787340    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg" event={"ID":"f9e786f9-5769-42b9-8b85-90507b4fdc5a","Type":"ContainerDied","Data":"c3b7b388fd542eb0ee42a7e81c08da50992032235c8b6cce59c7b9f5c7728317"}
Nov 02 02:00:17.787421 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:17.787380    5875 scope.go:122] "RemoveContainer" containerID="a8e2e1f3accb45b044ba42eb84b2e80345921d54c34d5e9f40f3bb1cdc5b4dd1"
Nov 02 02:00:17.801252 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:17.793736    5875 generic.go:356] "Generic (PLEG): container finished" podID="8ff0d1e0-810f-4091-92de-6e6575920d13" containerID="bb6ac0773cc6e2cedbe5fcc49b75d2c7058305ab6f8ad1b8e614a3696713da1b" exitCode=60
Nov 02 02:00:17.801252 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:17.793831    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874" event={"ID":"8ff0d1e0-810f-4091-92de-6e6575920d13","Type":"ContainerDied","Data":"bb6ac0773cc6e2cedbe5fcc49b75d2c7058305ab6f8ad1b8e614a3696713da1b"}
Nov 02 02:00:17.801252 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:17.794363    5875 scope.go:122] "RemoveContainer" containerID="bb6ac0773cc6e2cedbe5fcc49b75d2c7058305ab6f8ad1b8e614a3696713da1b"
Nov 02 02:00:17.801252 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:00:17.794751    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893080-d7874_odh-ai-gateway-infra(8ff0d1e0-810f-4091-92de-6e6575920d13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874" podUID="8ff0d1e0-810f-4091-92de-6e6575920d13"
Nov 02 02:00:17.801252 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:17.796607    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874"]
Nov 02 02:00:17.874258 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:17.873334    5875 patch_prober.go:29] interesting pod/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Nov 02 02:00:17.874258 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:17.873394    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf" podUID="d4b7a180-01b7-4c32-93be-dcb96b6b3b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Nov 02 02:00:17.874570 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:17.874486    5875 scope.go:122] "RemoveContainer" containerID="bfef1fffb9350441a209f78043052804a6bc89456e4cf487e61274fd43f9e747"
Nov 02 02:00:18.263523 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:18.263499    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg"
Nov 02 02:00:18.289579 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:18.289549    5875 patch_prober.go:29] interesting pod/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 02 02:00:18.289916 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:18.289598    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f" podUID="eedec700-8816-4b96-a087-7d0d691123bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 02 02:00:18.361366 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:18.361202    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9e786f9-5769-42b9-8b85-90507b4fdc5a-kube-api-access-sbh5c\" (UniqueName: \"kubernetes.io/projected/f9e786f9-5769-42b9-8b85-90507b4fdc5a-kube-api-access-sbh5c\") pod \"f9e786f9-5769-42b9-8b85-90507b4fdc5a\" (UID: \"f9e786f9-5769-42b9-8b85-90507b4fdc5a\") "
Nov 02 02:00:18.361700 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:18.361530    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9e786f9-5769-42b9-8b85-90507b4fdc5a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9e786f9-5769-42b9-8b85-90507b4fdc5a-maas-api-service-ca\") pod \"f9e786f9-5769-42b9-8b85-90507b4fdc5a\" (UID: \"f9e786f9-5769-42b9-8b85-90507b4fdc5a\") "
Nov 02 02:00:18.361965 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:18.361916    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9e786f9-5769-42b9-8b85-90507b4fdc5a-maas-api-service-ca" pod "f9e786f9-5769-42b9-8b85-90507b4fdc5a" (UID: "f9e786f9-5769-42b9-8b85-90507b4fdc5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:18.362749 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:18.362644    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9e786f9-5769-42b9-8b85-90507b4fdc5a-maas-api-service-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:18.365198 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:18.365172    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9e786f9-5769-42b9-8b85-90507b4fdc5a-kube-api-access-sbh5c" pod "f9e786f9-5769-42b9-8b85-90507b4fdc5a" (UID: "f9e786f9-5769-42b9-8b85-90507b4fdc5a"). InnerVolumeSpecName "kube-api-access-sbh5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:18.465013 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:18.464978    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbh5c\" (UniqueName: \"kubernetes.io/projected/f9e786f9-5769-42b9-8b85-90507b4fdc5a-kube-api-access-sbh5c\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:18.688047 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:18.688014    5875 scope.go:122] "RemoveContainer" containerID="2e541361724db20ea201e0d027ee1490aef0234591d6eb28ef5e264dc36f312a"
Nov 02 02:00:18.818036 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:18.817994    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f" event={"ID":"3fdb61ec-f124-4f24-9ac4-94256e0ab23d","Type":"ContainerStarted","Data":"c200c276424e031a8f7f5282f9978137588aea276b2b6ae3b5ad67c3049d04bf"}
Nov 02 02:00:18.819957 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:18.819935    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg"
Nov 02 02:00:18.820087 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:18.819956    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg" event={"ID":"f9e786f9-5769-42b9-8b85-90507b4fdc5a","Type":"ContainerDied","Data":"cc64e91e84b33a659c87c6fdbfec41aefcd8379c4359b56ebd3f4ba9de9a09f7"}
Nov 02 02:00:18.820087 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:18.820043    5875 scope.go:122] "RemoveContainer" containerID="c3b7b388fd542eb0ee42a7e81c08da50992032235c8b6cce59c7b9f5c7728317"
Nov 02 02:00:18.822729 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:18.822700    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k" event={"ID":"927e4a13-38c2-4b2c-a630-35db116a5ed3","Type":"ContainerStarted","Data":"74e35f324ccb559b9b9c355ea35d802e12173abcc73fa3fb7863f26eba42ac47"}
Nov 02 02:00:18.874257 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:18.874132    5875 patch_prober.go:29] interesting pod/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Nov 02 02:00:18.874257 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:18.874207    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf" podUID="d4b7a180-01b7-4c32-93be-dcb96b6b3b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Nov 02 02:00:19.132875 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.132855    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874"
Nov 02 02:00:19.166377 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.166349    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg"]
Nov 02 02:00:19.171274 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.171255    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ed0600-29893080-47fpg"]
Nov 02 02:00:19.199496 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.199473    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ff0d1e0-810f-4091-92de-6e6575920d13-kube-api-access-x4qqv\" (UniqueName: \"kubernetes.io/projected/8ff0d1e0-810f-4091-92de-6e6575920d13-kube-api-access-x4qqv\") pod \"8ff0d1e0-810f-4091-92de-6e6575920d13\" (UID: \"8ff0d1e0-810f-4091-92de-6e6575920d13\") "
Nov 02 02:00:19.199675 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.199660    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ff0d1e0-810f-4091-92de-6e6575920d13-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff0d1e0-810f-4091-92de-6e6575920d13-maas-api-service-ca\") pod \"8ff0d1e0-810f-4091-92de-6e6575920d13\" (UID: \"8ff0d1e0-810f-4091-92de-6e6575920d13\") "
Nov 02 02:00:19.200010 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.199983    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ff0d1e0-810f-4091-92de-6e6575920d13-maas-api-service-ca" pod "8ff0d1e0-810f-4091-92de-6e6575920d13" (UID: "8ff0d1e0-810f-4091-92de-6e6575920d13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:19.200384 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.200366    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff0d1e0-810f-4091-92de-6e6575920d13-maas-api-service-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:19.201760 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.201736    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff0d1e0-810f-4091-92de-6e6575920d13-kube-api-access-x4qqv" pod "8ff0d1e0-810f-4091-92de-6e6575920d13" (UID: "8ff0d1e0-810f-4091-92de-6e6575920d13"). InnerVolumeSpecName "kube-api-access-x4qqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:19.293893 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.293826    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 02:00:19.294329 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.294124    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 02:00:19.294911 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.294892    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 02:00:19.301908 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.301885    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4qqv\" (UniqueName: \"kubernetes.io/projected/8ff0d1e0-810f-4091-92de-6e6575920d13-kube-api-access-x4qqv\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:19.832518 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.832488    5875 generic.go:356] "Generic (PLEG): container finished" podID="3fdb61ec-f124-4f24-9ac4-94256e0ab23d" containerID="c200c276424e031a8f7f5282f9978137588aea276b2b6ae3b5ad67c3049d04bf" exitCode=60
Nov 02 02:00:19.832622 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.832557    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f" event={"ID":"3fdb61ec-f124-4f24-9ac4-94256e0ab23d","Type":"ContainerDied","Data":"c200c276424e031a8f7f5282f9978137588aea276b2b6ae3b5ad67c3049d04bf"}
Nov 02 02:00:19.832622 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.832618    5875 scope.go:122] "RemoveContainer" containerID="08000c17a58300270b8a46e57944a1d39488b7ab7a977afa5602eaac118e03fd"
Nov 02 02:00:19.833059 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.833040    5875 scope.go:122] "RemoveContainer" containerID="c200c276424e031a8f7f5282f9978137588aea276b2b6ae3b5ad67c3049d04bf"
Nov 02 02:00:19.833479 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:00:19.833453    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f_odh-ai-gateway-infra(3fdb61ec-f124-4f24-9ac4-94256e0ab23d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f" podUID="3fdb61ec-f124-4f24-9ac4-94256e0ab23d"
Nov 02 02:00:19.835826 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.835802    5875 generic.go:356] "Generic (PLEG): container finished" podID="927e4a13-38c2-4b2c-a630-35db116a5ed3" containerID="74e35f324ccb559b9b9c355ea35d802e12173abcc73fa3fb7863f26eba42ac47" exitCode=60
Nov 02 02:00:19.835928 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.835824    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k" event={"ID":"927e4a13-38c2-4b2c-a630-35db116a5ed3","Type":"ContainerDied","Data":"74e35f324ccb559b9b9c355ea35d802e12173abcc73fa3fb7863f26eba42ac47"}
Nov 02 02:00:19.836270 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.836250    5875 scope.go:122] "RemoveContainer" containerID="74e35f324ccb559b9b9c355ea35d802e12173abcc73fa3fb7863f26eba42ac47"
Nov 02 02:00:19.836595 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:00:19.836570    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k_odh-ai-gateway-infra(927e4a13-38c2-4b2c-a630-35db116a5ed3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k" podUID="927e4a13-38c2-4b2c-a630-35db116a5ed3"
Nov 02 02:00:19.837591 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.837578    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874"
Nov 02 02:00:19.837677 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.837615    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874" event={"ID":"8ff0d1e0-810f-4091-92de-6e6575920d13","Type":"ContainerDied","Data":"7779e49e4db2fadd16a375f976eea0db2e7db73710a43e01198345995e9e2931"}
Nov 02 02:00:19.839611 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.839590    5875 generic.go:356] "Generic (PLEG): container finished" podID="025b1e27-ad55-46ed-a362-ab9e23c4d6d5" containerID="53d9ddea40da063b110cce05b249ec8663240e5b604a5a92335a7fbeee86f4fb" exitCode=60
Nov 02 02:00:19.839702 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.839662    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh" event={"ID":"025b1e27-ad55-46ed-a362-ab9e23c4d6d5","Type":"ContainerDied","Data":"53d9ddea40da063b110cce05b249ec8663240e5b604a5a92335a7fbeee86f4fb"}
Nov 02 02:00:19.840058 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.840046    5875 scope.go:122] "RemoveContainer" containerID="53d9ddea40da063b110cce05b249ec8663240e5b604a5a92335a7fbeee86f4fb"
Nov 02 02:00:19.840349 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:00:19.840331    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh_odh-ai-gateway-infra(025b1e27-ad55-46ed-a362-ab9e23c4d6d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh" podUID="025b1e27-ad55-46ed-a362-ab9e23c4d6d5"
Nov 02 02:00:19.869051 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.869026    5875 scope.go:122] "RemoveContainer" containerID="5d92e21f69fd4c2a8890928cc7e1a1d3578c83c412161b70c147742dada55695"
Nov 02 02:00:19.879411 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.879388    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 02:00:19.880534 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.880514    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 02:00:19.882635 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.882620    5875 scope.go:122] "RemoveContainer" containerID="bb6ac0773cc6e2cedbe5fcc49b75d2c7058305ab6f8ad1b8e614a3696713da1b"
Nov 02 02:00:19.892345 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:19.892325    5875 scope.go:122] "RemoveContainer" containerID="2e541361724db20ea201e0d027ee1490aef0234591d6eb28ef5e264dc36f312a"
Nov 02 02:00:20.063706 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:20.063623    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f"]
Nov 02 02:00:20.305879 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:20.305784    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k"]
Nov 02 02:00:20.314271 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:20.314144    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874"]
Nov 02 02:00:20.318693 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:20.318661    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893080-d7874"]
Nov 02 02:00:20.697809 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:20.697727    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ff0d1e0-810f-4091-92de-6e6575920d13" path="/var/lib/kubelet/pods/8ff0d1e0-810f-4091-92de-6e6575920d13/volumes"
Nov 02 02:00:20.698292 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:20.698277    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9e786f9-5769-42b9-8b85-90507b4fdc5a" path="/var/lib/kubelet/pods/f9e786f9-5769-42b9-8b85-90507b4fdc5a/volumes"
Nov 02 02:00:21.048639 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.048608    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f"
Nov 02 02:00:21.107800 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.107773    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k"
Nov 02 02:00:21.135024 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.133182    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/927e4a13-38c2-4b2c-a630-35db116a5ed3-kube-api-access-wj7fl\" (UniqueName: \"kubernetes.io/projected/927e4a13-38c2-4b2c-a630-35db116a5ed3-kube-api-access-wj7fl\") pod \"927e4a13-38c2-4b2c-a630-35db116a5ed3\" (UID: \"927e4a13-38c2-4b2c-a630-35db116a5ed3\") "
Nov 02 02:00:21.135024 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.133334    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/927e4a13-38c2-4b2c-a630-35db116a5ed3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/927e4a13-38c2-4b2c-a630-35db116a5ed3-maas-api-service-ca\") pod \"927e4a13-38c2-4b2c-a630-35db116a5ed3\" (UID: \"927e4a13-38c2-4b2c-a630-35db116a5ed3\") "
Nov 02 02:00:21.135024 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.133485    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fdb61ec-f124-4f24-9ac4-94256e0ab23d-kube-api-access-hggvz\" (UniqueName: \"kubernetes.io/projected/3fdb61ec-f124-4f24-9ac4-94256e0ab23d-kube-api-access-hggvz\") pod \"3fdb61ec-f124-4f24-9ac4-94256e0ab23d\" (UID: \"3fdb61ec-f124-4f24-9ac4-94256e0ab23d\") "
Nov 02 02:00:21.135024 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.133687    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fdb61ec-f124-4f24-9ac4-94256e0ab23d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdb61ec-f124-4f24-9ac4-94256e0ab23d-maas-api-service-ca\") pod \"3fdb61ec-f124-4f24-9ac4-94256e0ab23d\" (UID: \"3fdb61ec-f124-4f24-9ac4-94256e0ab23d\") "
Nov 02 02:00:21.135024 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.133769    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/927e4a13-38c2-4b2c-a630-35db116a5ed3-maas-api-service-ca" pod "927e4a13-38c2-4b2c-a630-35db116a5ed3" (UID: "927e4a13-38c2-4b2c-a630-35db116a5ed3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:21.135024 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.134546    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fdb61ec-f124-4f24-9ac4-94256e0ab23d-maas-api-service-ca" pod "3fdb61ec-f124-4f24-9ac4-94256e0ab23d" (UID: "3fdb61ec-f124-4f24-9ac4-94256e0ab23d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:21.135024 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.134723    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/927e4a13-38c2-4b2c-a630-35db116a5ed3-maas-api-service-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:21.135024 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.134743    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdb61ec-f124-4f24-9ac4-94256e0ab23d-maas-api-service-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:21.136574 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.136552    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fdb61ec-f124-4f24-9ac4-94256e0ab23d-kube-api-access-hggvz" pod "3fdb61ec-f124-4f24-9ac4-94256e0ab23d" (UID: "3fdb61ec-f124-4f24-9ac4-94256e0ab23d"). InnerVolumeSpecName "kube-api-access-hggvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:21.136674 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.136589    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/927e4a13-38c2-4b2c-a630-35db116a5ed3-kube-api-access-wj7fl" pod "927e4a13-38c2-4b2c-a630-35db116a5ed3" (UID: "927e4a13-38c2-4b2c-a630-35db116a5ed3"). InnerVolumeSpecName "kube-api-access-wj7fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:21.247248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.247118    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj7fl\" (UniqueName: \"kubernetes.io/projected/927e4a13-38c2-4b2c-a630-35db116a5ed3-kube-api-access-wj7fl\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:21.247248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.247161    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hggvz\" (UniqueName: \"kubernetes.io/projected/3fdb61ec-f124-4f24-9ac4-94256e0ab23d-kube-api-access-hggvz\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:21.429384 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.429353    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh"]
Nov 02 02:00:21.580089 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.580061    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh"
Nov 02 02:00:21.652418 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.652392    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/025b1e27-ad55-46ed-a362-ab9e23c4d6d5-kube-api-access-92whj\" (UniqueName: \"kubernetes.io/projected/025b1e27-ad55-46ed-a362-ab9e23c4d6d5-kube-api-access-92whj\") pod \"025b1e27-ad55-46ed-a362-ab9e23c4d6d5\" (UID: \"025b1e27-ad55-46ed-a362-ab9e23c4d6d5\") "
Nov 02 02:00:21.652569 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.652438    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/025b1e27-ad55-46ed-a362-ab9e23c4d6d5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/025b1e27-ad55-46ed-a362-ab9e23c4d6d5-maas-api-service-ca\") pod \"025b1e27-ad55-46ed-a362-ab9e23c4d6d5\" (UID: \"025b1e27-ad55-46ed-a362-ab9e23c4d6d5\") "
Nov 02 02:00:21.652833 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.652809    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/025b1e27-ad55-46ed-a362-ab9e23c4d6d5-maas-api-service-ca" pod "025b1e27-ad55-46ed-a362-ab9e23c4d6d5" (UID: "025b1e27-ad55-46ed-a362-ab9e23c4d6d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:21.653134 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.653117    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/025b1e27-ad55-46ed-a362-ab9e23c4d6d5-maas-api-service-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:21.654604 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.654582    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025b1e27-ad55-46ed-a362-ab9e23c4d6d5-kube-api-access-92whj" pod "025b1e27-ad55-46ed-a362-ab9e23c4d6d5" (UID: "025b1e27-ad55-46ed-a362-ab9e23c4d6d5"). InnerVolumeSpecName "kube-api-access-92whj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:21.754987 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.754941    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92whj\" (UniqueName: \"kubernetes.io/projected/025b1e27-ad55-46ed-a362-ab9e23c4d6d5-kube-api-access-92whj\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:21.863415 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.863342    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f"
Nov 02 02:00:21.863415 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.863381    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f" event={"ID":"3fdb61ec-f124-4f24-9ac4-94256e0ab23d","Type":"ContainerDied","Data":"815aae56282273a73dda60c956bda186e668689b8b85a1706fb87964d5c1e629"}
Nov 02 02:00:21.863643 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.863447    5875 scope.go:122] "RemoveContainer" containerID="c200c276424e031a8f7f5282f9978137588aea276b2b6ae3b5ad67c3049d04bf"
Nov 02 02:00:21.865036 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.865017    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k"
Nov 02 02:00:21.865129 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.865039    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k" event={"ID":"927e4a13-38c2-4b2c-a630-35db116a5ed3","Type":"ContainerDied","Data":"554b390609f614ba3122e7499d9b24cba2c54b9ebfcc58254caec4d67f0445ac"}
Nov 02 02:00:21.866884 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.866862    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh" event={"ID":"025b1e27-ad55-46ed-a362-ab9e23c4d6d5","Type":"ContainerDied","Data":"f7e5cb01d75579d89158f508d8e9c99b4b23f5161916ca9125e9fc8a9853ca0b"}
Nov 02 02:00:21.866996 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.866906    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh"
Nov 02 02:00:21.875780 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.875765    5875 scope.go:122] "RemoveContainer" containerID="74e35f324ccb559b9b9c355ea35d802e12173abcc73fa3fb7863f26eba42ac47"
Nov 02 02:00:21.889036 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:21.889010    5875 scope.go:122] "RemoveContainer" containerID="53d9ddea40da063b110cce05b249ec8663240e5b604a5a92335a7fbeee86f4fb"
Nov 02 02:00:22.028998 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:22.028964    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh"]
Nov 02 02:00:22.033066 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:22.033042    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893080-hmsgh"]
Nov 02 02:00:22.139996 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:22.139917    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f"]
Nov 02 02:00:22.145014 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:22.144991    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-061fab-29893080-6vc7f"]
Nov 02 02:00:22.252843 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:22.252797    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k"]
Nov 02 02:00:22.256734 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:22.256712    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4f2f926d-29893080-85r8k"]
Nov 02 02:00:22.696497 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:22.696468    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="025b1e27-ad55-46ed-a362-ab9e23c4d6d5" path="/var/lib/kubelet/pods/025b1e27-ad55-46ed-a362-ab9e23c4d6d5/volumes"
Nov 02 02:00:22.696906 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:22.696892    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fdb61ec-f124-4f24-9ac4-94256e0ab23d" path="/var/lib/kubelet/pods/3fdb61ec-f124-4f24-9ac4-94256e0ab23d/volumes"
Nov 02 02:00:22.697335 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:22.697322    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="927e4a13-38c2-4b2c-a630-35db116a5ed3" path="/var/lib/kubelet/pods/927e4a13-38c2-4b2c-a630-35db116a5ed3/volumes"
Nov 02 02:00:26.277928 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:26.277895    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"]
Nov 02 02:00:26.278437 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:26.278089    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6" podUID="a1c34e84-6fd7-43d7-9db1-7a9e755c136d" containerName="istio-proxy" containerID="cri-o://ae935c11e54ee0c3d3304bf14d1c855cbe172ab6e3b41e64d4c8bd659050eb99" gracePeriod=30
Nov 02 02:00:26.463581 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:26.463544    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2"]
Nov 02 02:00:26.463901 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:26.463844    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2" podUID="26ea638a-9657-45c0-905a-c04ef3d7358f" containerName="maas-api" containerID="cri-o://7c17640f22e25649cc7a26ab008dcb674b21fcbe1d0684b1635f0871990548a7" gracePeriod=30
Nov 02 02:00:27.181394 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:27.181357    5875 generic.go:356] "Generic (PLEG): container finished" podID="26ea638a-9657-45c0-905a-c04ef3d7358f" containerID="7c17640f22e25649cc7a26ab008dcb674b21fcbe1d0684b1635f0871990548a7" exitCode=0
Nov 02 02:00:27.181714 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:27.181694    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2" event={"ID":"26ea638a-9657-45c0-905a-c04ef3d7358f","Type":"ContainerDied","Data":"7c17640f22e25649cc7a26ab008dcb674b21fcbe1d0684b1635f0871990548a7"}
Nov 02 02:00:27.210816 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:27.210792    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2"
Nov 02 02:00:27.302785 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:27.302758    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26ea638a-9657-45c0-905a-c04ef3d7358f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ea638a-9657-45c0-905a-c04ef3d7358f-openshift-service-ca\") pod \"26ea638a-9657-45c0-905a-c04ef3d7358f\" (UID: \"26ea638a-9657-45c0-905a-c04ef3d7358f\") "
Nov 02 02:00:27.303199 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:27.302816    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26ea638a-9657-45c0-905a-c04ef3d7358f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ea638a-9657-45c0-905a-c04ef3d7358f-maas-api-tls\") pod \"26ea638a-9657-45c0-905a-c04ef3d7358f\" (UID: \"26ea638a-9657-45c0-905a-c04ef3d7358f\") "
Nov 02 02:00:27.303199 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:27.302982    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ea638a-9657-45c0-905a-c04ef3d7358f-kube-api-access-w7nxp\" (UniqueName: \"kubernetes.io/projected/26ea638a-9657-45c0-905a-c04ef3d7358f-kube-api-access-w7nxp\") pod \"26ea638a-9657-45c0-905a-c04ef3d7358f\" (UID: \"26ea638a-9657-45c0-905a-c04ef3d7358f\") "
Nov 02 02:00:27.303199 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:27.303036    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26ea638a-9657-45c0-905a-c04ef3d7358f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ea638a-9657-45c0-905a-c04ef3d7358f-metrics-certs\") pod \"26ea638a-9657-45c0-905a-c04ef3d7358f\" (UID: \"26ea638a-9657-45c0-905a-c04ef3d7358f\") "
Nov 02 02:00:27.303199 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:27.303099    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ea638a-9657-45c0-905a-c04ef3d7358f-openshift-service-ca" pod "26ea638a-9657-45c0-905a-c04ef3d7358f" (UID: "26ea638a-9657-45c0-905a-c04ef3d7358f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:27.304071 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:27.304047    5875 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ea638a-9657-45c0-905a-c04ef3d7358f-openshift-service-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:27.305303 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:27.305274    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26ea638a-9657-45c0-905a-c04ef3d7358f-metrics-certs" pod "26ea638a-9657-45c0-905a-c04ef3d7358f" (UID: "26ea638a-9657-45c0-905a-c04ef3d7358f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:00:27.305390 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:27.305317    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ea638a-9657-45c0-905a-c04ef3d7358f-kube-api-access-w7nxp" pod "26ea638a-9657-45c0-905a-c04ef3d7358f" (UID: "26ea638a-9657-45c0-905a-c04ef3d7358f"). InnerVolumeSpecName "kube-api-access-w7nxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:27.305390 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:27.305372    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26ea638a-9657-45c0-905a-c04ef3d7358f-maas-api-tls" pod "26ea638a-9657-45c0-905a-c04ef3d7358f" (UID: "26ea638a-9657-45c0-905a-c04ef3d7358f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:00:27.405340 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:27.405269    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ea638a-9657-45c0-905a-c04ef3d7358f-maas-api-tls\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:27.405340 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:27.405295    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7nxp\" (UniqueName: \"kubernetes.io/projected/26ea638a-9657-45c0-905a-c04ef3d7358f-kube-api-access-w7nxp\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:27.405340 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:27.405307    5875 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ea638a-9657-45c0-905a-c04ef3d7358f-metrics-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:28.191167 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:28.191143    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2"
Nov 02 02:00:28.191367 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:28.191137    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2" event={"ID":"26ea638a-9657-45c0-905a-c04ef3d7358f","Type":"ContainerDied","Data":"f162af31163eb580a10ffbe6abf9dd2761b2681888ae070c98a3278a0ecec279"}
Nov 02 02:00:28.191367 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:28.191310    5875 scope.go:122] "RemoveContainer" containerID="7c17640f22e25649cc7a26ab008dcb674b21fcbe1d0684b1635f0871990548a7"
Nov 02 02:00:28.356439 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:28.356410    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2"]
Nov 02 02:00:28.406235 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:28.406193    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4f2f926d-58f644df4-g9jx2"]
Nov 02 02:00:28.744252 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:28.742774    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26ea638a-9657-45c0-905a-c04ef3d7358f" path="/var/lib/kubelet/pods/26ea638a-9657-45c0-905a-c04ef3d7358f/volumes"
Nov 02 02:00:30.335432 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:30.335403    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"]
Nov 02 02:00:30.335935 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:30.335610    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz" podUID="98296768-250b-4160-8678-b92e2fb3a151" containerName="istio-proxy" containerID="cri-o://cd64b3f296c66d4c5bc3bd3c63362e50d740a4b6b8c0e63fd116ee4500f86f80" gracePeriod=30
Nov 02 02:00:31.251073 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.251038    5875 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:00:31.251293 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.251096    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6" podUID="a1c34e84-6fd7-43d7-9db1-7a9e755c136d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:00:31.625961 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.625937    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 02:00:31.778442 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.778400    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"]
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794523    5875 cpu_manager.go:414] "containerMap: removing container" podUID="3fdb61ec-f124-4f24-9ac4-94256e0ab23d" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794557    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fdb61ec-f124-4f24-9ac4-94256e0ab23d" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794581    5875 cpu_manager.go:414] "containerMap: removing container" podUID="025b1e27-ad55-46ed-a362-ab9e23c4d6d5" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794590    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025b1e27-ad55-46ed-a362-ab9e23c4d6d5" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794616    5875 cpu_manager.go:414] "containerMap: removing container" podUID="927e4a13-38c2-4b2c-a630-35db116a5ed3" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794624    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="927e4a13-38c2-4b2c-a630-35db116a5ed3" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794643    5875 cpu_manager.go:414] "containerMap: removing container" podUID="8ff0d1e0-810f-4091-92de-6e6575920d13" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794651    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ff0d1e0-810f-4091-92de-6e6575920d13" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794677    5875 cpu_manager.go:414] "containerMap: removing container" podUID="8ff0d1e0-810f-4091-92de-6e6575920d13" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794684    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ff0d1e0-810f-4091-92de-6e6575920d13" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794696    5875 cpu_manager.go:414] "containerMap: removing container" podUID="3fdb61ec-f124-4f24-9ac4-94256e0ab23d" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794703    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fdb61ec-f124-4f24-9ac4-94256e0ab23d" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794721    5875 cpu_manager.go:414] "containerMap: removing container" podUID="f9e786f9-5769-42b9-8b85-90507b4fdc5a" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794728    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9e786f9-5769-42b9-8b85-90507b4fdc5a" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794739    5875 cpu_manager.go:414] "containerMap: removing container" podUID="f9e786f9-5769-42b9-8b85-90507b4fdc5a" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794746    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9e786f9-5769-42b9-8b85-90507b4fdc5a" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794775    5875 cpu_manager.go:414] "containerMap: removing container" podUID="f9e786f9-5769-42b9-8b85-90507b4fdc5a" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794782    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9e786f9-5769-42b9-8b85-90507b4fdc5a" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794805    5875 cpu_manager.go:414] "containerMap: removing container" podUID="025b1e27-ad55-46ed-a362-ab9e23c4d6d5" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794812    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025b1e27-ad55-46ed-a362-ab9e23c4d6d5" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794830    5875 cpu_manager.go:414] "containerMap: removing container" podUID="927e4a13-38c2-4b2c-a630-35db116a5ed3" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794836    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="927e4a13-38c2-4b2c-a630-35db116a5ed3" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794849    5875 cpu_manager.go:414] "containerMap: removing container" podUID="a1c34e84-6fd7-43d7-9db1-7a9e755c136d" containerName="istio-proxy"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794857    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c34e84-6fd7-43d7-9db1-7a9e755c136d" containerName="istio-proxy"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794866    5875 cpu_manager.go:414] "containerMap: removing container" podUID="26ea638a-9657-45c0-905a-c04ef3d7358f" containerName="maas-api"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.794873    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ea638a-9657-45c0-905a-c04ef3d7358f" containerName="maas-api"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.795068    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ff0d1e0-810f-4091-92de-6e6575920d13" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.795082    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fdb61ec-f124-4f24-9ac4-94256e0ab23d" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.795092    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9e786f9-5769-42b9-8b85-90507b4fdc5a" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.795104    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9e786f9-5769-42b9-8b85-90507b4fdc5a" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.795115    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="927e4a13-38c2-4b2c-a630-35db116a5ed3" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.795128    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9e786f9-5769-42b9-8b85-90507b4fdc5a" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.795139    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ff0d1e0-810f-4091-92de-6e6575920d13" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.795147    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="025b1e27-ad55-46ed-a362-ab9e23c4d6d5" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.795154    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="025b1e27-ad55-46ed-a362-ab9e23c4d6d5" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.795164    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="927e4a13-38c2-4b2c-a630-35db116a5ed3" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.795173    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c34e84-6fd7-43d7-9db1-7a9e755c136d" containerName="istio-proxy"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.795180    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ea638a-9657-45c0-905a-c04ef3d7358f" containerName="maas-api"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.795193    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fdb61ec-f124-4f24-9ac4-94256e0ab23d" containerName="cleanup"
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.817977    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istiod-ca-cert\") pod \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") "
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.818086    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-podinfo\") pod \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") "
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.818252    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-data\") pod \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") "
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.818339    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-envoy\") pod \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") "
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.818413    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-credential-socket\") pod \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") "
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.818499    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-workload-certs\") pod \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") "
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.818635    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-kube-api-access-nthsl\" (UniqueName: \"kubernetes.io/projected/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-kube-api-access-nthsl\") pod \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") "
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.818773    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-token\" (UniqueName: \"kubernetes.io/projected/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-token\") pod \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") "
Nov 02 02:00:31.820251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.818823    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-workload-socket\") pod \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\" (UID: \"a1c34e84-6fd7-43d7-9db1-7a9e755c136d\") "
Nov 02 02:00:31.964815 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.845345    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-workload-certs" pod "a1c34e84-6fd7-43d7-9db1-7a9e755c136d" (UID: "a1c34e84-6fd7-43d7-9db1-7a9e755c136d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:31.964815 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.849376    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-credential-socket" pod "a1c34e84-6fd7-43d7-9db1-7a9e755c136d" (UID: "a1c34e84-6fd7-43d7-9db1-7a9e755c136d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:31.964815 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.890283    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:31.964815 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.890314    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:31.964815 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.925331    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-podinfo" pod "a1c34e84-6fd7-43d7-9db1-7a9e755c136d" (UID: "a1c34e84-6fd7-43d7-9db1-7a9e755c136d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:00:31.964815 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.955359    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-workload-socket" pod "a1c34e84-6fd7-43d7-9db1-7a9e755c136d" (UID: "a1c34e84-6fd7-43d7-9db1-7a9e755c136d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:31.964815 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.955568    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istiod-ca-cert" pod "a1c34e84-6fd7-43d7-9db1-7a9e755c136d" (UID: "a1c34e84-6fd7-43d7-9db1-7a9e755c136d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:31.995611 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.995572    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-data" pod "a1c34e84-6fd7-43d7-9db1-7a9e755c136d" (UID: "a1c34e84-6fd7-43d7-9db1-7a9e755c136d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:31.995889 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.995838    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:31.996141 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.996118    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:31.996307 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.996291    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:32.018242 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.999407    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"]
Nov 02 02:00:32.018242 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:31.999525    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.018242 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.015068    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-kube-api-access-nthsl" pod "a1c34e84-6fd7-43d7-9db1-7a9e755c136d" (UID: "a1c34e84-6fd7-43d7-9db1-7a9e755c136d"). InnerVolumeSpecName "kube-api-access-nthsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:32.047501 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.028421    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-token" pod "a1c34e84-6fd7-43d7-9db1-7a9e755c136d" (UID: "a1c34e84-6fd7-43d7-9db1-7a9e755c136d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:32.047501 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.028517    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-envoy" pod "a1c34e84-6fd7-43d7-9db1-7a9e755c136d" (UID: "a1c34e84-6fd7-43d7-9db1-7a9e755c136d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:32.122898 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.106118    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kczh\" (UniqueName: \"kubernetes.io/projected/a0c5779f-1d48-4749-bcb0-8313b202142d-kube-api-access-6kczh\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.122898 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.106280    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0c5779f-1d48-4749-bcb0-8313b202142d-istiod-ca-cert\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.122898 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.106336    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-workload-certs\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.122898 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.106404    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-workload-socket\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.178428 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.127532    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-129de4-openshift-default-dockercfg-nf4cx\""
Nov 02 02:00:32.178428 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.134738    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-credential-socket\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.178428 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.134849    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-token\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.178428 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.135050    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-envoy\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.178428 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.135199    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-data\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.178428 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.135289    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-podinfo\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.178428 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.141162    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:32.178428 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.141195    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:32.178428 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.141242    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:32.178428 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.141260    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nthsl\" (UniqueName: \"kubernetes.io/projected/a1c34e84-6fd7-43d7-9db1-7a9e755c136d-kube-api-access-nthsl\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:32.255306 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.245580    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-credential-socket\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.256016 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.255991    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-token\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.256744 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.256722    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-envoy\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.260212 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.260190    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-data\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.260728 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.260700    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-podinfo\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.261389 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.261368    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kczh\" (UniqueName: \"kubernetes.io/projected/a0c5779f-1d48-4749-bcb0-8313b202142d-kube-api-access-6kczh\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.262036 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.262016    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0c5779f-1d48-4749-bcb0-8313b202142d-istiod-ca-cert\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.262937 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.262919    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-workload-certs\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.263442 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.263424    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-workload-socket\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.264139 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.255920    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-credential-socket\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.264290 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.263319    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-workload-certs\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.264415 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.260636    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-data\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.264506 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.262792    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0c5779f-1d48-4749-bcb0-8313b202142d-istiod-ca-cert\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.264602 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.263860    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-workload-socket\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.343516 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.321870    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-token\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.343516 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.325903    5875 generic.go:356] "Generic (PLEG): container finished" podID="a1c34e84-6fd7-43d7-9db1-7a9e755c136d" containerID="ae935c11e54ee0c3d3304bf14d1c855cbe172ab6e3b41e64d4c8bd659050eb99" exitCode=0
Nov 02 02:00:32.343516 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.325969    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6" event={"ID":"a1c34e84-6fd7-43d7-9db1-7a9e755c136d","Type":"ContainerDied","Data":"ae935c11e54ee0c3d3304bf14d1c855cbe172ab6e3b41e64d4c8bd659050eb99"}
Nov 02 02:00:32.343516 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.326012    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6" event={"ID":"a1c34e84-6fd7-43d7-9db1-7a9e755c136d","Type":"ContainerDied","Data":"4c1419144f2b724bcca6fd136943c43f6ca3eabda910fd0f11bb3b3f4921e766"}
Nov 02 02:00:32.343516 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.326037    5875 scope.go:122] "RemoveContainer" containerID="ae935c11e54ee0c3d3304bf14d1c855cbe172ab6e3b41e64d4c8bd659050eb99"
Nov 02 02:00:32.343516 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.326149    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"
Nov 02 02:00:32.345873 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.345815    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-envoy\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.352253 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.348838    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-podinfo\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.352253 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.352060    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kczh\" (UniqueName: \"kubernetes.io/projected/a0c5779f-1d48-4749-bcb0-8313b202142d-kube-api-access-6kczh\") pod \"e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") " pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.357447 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.357133    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:32.409402 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.409370    5875 scope.go:122] "RemoveContainer" containerID="ae935c11e54ee0c3d3304bf14d1c855cbe172ab6e3b41e64d4c8bd659050eb99"
Nov 02 02:00:32.409805 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:00:32.409783    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae935c11e54ee0c3d3304bf14d1c855cbe172ab6e3b41e64d4c8bd659050eb99\": container with ID starting with ae935c11e54ee0c3d3304bf14d1c855cbe172ab6e3b41e64d4c8bd659050eb99 not found: ID does not exist" containerID="ae935c11e54ee0c3d3304bf14d1c855cbe172ab6e3b41e64d4c8bd659050eb99"
Nov 02 02:00:32.409891 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.409818    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae935c11e54ee0c3d3304bf14d1c855cbe172ab6e3b41e64d4c8bd659050eb99"} err="failed to get container status \"ae935c11e54ee0c3d3304bf14d1c855cbe172ab6e3b41e64d4c8bd659050eb99\": rpc error: code = NotFound desc = could not find container \"ae935c11e54ee0c3d3304bf14d1c855cbe172ab6e3b41e64d4c8bd659050eb99\": container with ID starting with ae935c11e54ee0c3d3304bf14d1c855cbe172ab6e3b41e64d4c8bd659050eb99 not found: ID does not exist"
Nov 02 02:00:32.518884 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.518802    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"]
Nov 02 02:00:32.523172 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.523147    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-7dfd6"]
Nov 02 02:00:32.697020 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.696990    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1c34e84-6fd7-43d7-9db1-7a9e755c136d" path="/var/lib/kubelet/pods/a1c34e84-6fd7-43d7-9db1-7a9e755c136d/volumes"
Nov 02 02:00:32.829725 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.829696    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"]
Nov 02 02:00:32.830428 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:00:32.830401    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c5779f_1d48_4749_bcb0_8313b202142d.slice/crio-a7ae64c9939b0731e45a454ef3650c5f6d3c802a9c2bfef6a26b7caee9ab0c4b WatchSource:0}: Error finding container a7ae64c9939b0731e45a454ef3650c5f6d3c802a9c2bfef6a26b7caee9ab0c4b: Status 404 returned error can't find the container with id a7ae64c9939b0731e45a454ef3650c5f6d3c802a9c2bfef6a26b7caee9ab0c4b
Nov 02 02:00:32.833368 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.833336    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:00:32.833447 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.833401    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:00:32.833447 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:32.833429    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:00:33.420696 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:33.420656    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t" event={"ID":"a0c5779f-1d48-4749-bcb0-8313b202142d","Type":"ContainerStarted","Data":"547d14704548a99f3eaf569e2fe5da0ee2b16399bd0d050fd3ea68cee2438cdf"}
Nov 02 02:00:33.420974 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:33.420955    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t" event={"ID":"a0c5779f-1d48-4749-bcb0-8313b202142d","Type":"ContainerStarted","Data":"a7ae64c9939b0731e45a454ef3650c5f6d3c802a9c2bfef6a26b7caee9ab0c4b"}
Nov 02 02:00:33.614682 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:33.614639    5875 patch_prober.go:29] interesting pod/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:00:33.614874 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:33.614701    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz" podUID="98296768-250b-4160-8678-b92e2fb3a151" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:00:34.307679 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:34.306422    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t" podStartSLOduration=3.305234088 podStartE2EDuration="3.305234088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:33.614604768 +0000 UTC m=+1681.055641264" watchObservedRunningTime="2026-11-02 02:00:34.305234088 +0000 UTC m=+1681.746270579"
Nov 02 02:00:34.307679 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:34.307153    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:00:34.358264 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:34.358233    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:34.363903 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:34.363877    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:34.439349 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:34.439312    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:34.440289 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:34.440263    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:00:34.836359 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:34.836320    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"]
Nov 02 02:00:34.837090 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:34.837054    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g" podUID="0c95a482-2bd1-457e-9f4a-facf829b7699" containerName="istio-proxy" containerID="cri-o://d9d62939dac7d40b7165cca55ac30e42ffa615168fcfae9e04fa1bf8c0de3dc6" gracePeriod=30
Nov 02 02:00:34.845096 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:34.844679    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"]
Nov 02 02:00:34.845096 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:34.844912    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz" podUID="752ab42e-3d04-4340-8481-9592789c0367" containerName="istio-proxy" containerID="cri-o://153ec1dfc61d7fa222cc86a5f48663a9b67b0a75fe80349c483e785922ab015d" gracePeriod=30
Nov 02 02:00:34.869670 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:34.869628    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"]
Nov 02 02:00:34.870281 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:34.870205    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk" podUID="9e5f896f-d35c-4cbe-87e2-f18aa35b5a37" containerName="istio-proxy" containerID="cri-o://5fd885597ea0f87801233ad4c73a12efc8774735e406ba072a4100d700fc9106" gracePeriod=30
Nov 02 02:00:34.887804 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:34.887257    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"]
Nov 02 02:00:34.887804 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:34.887548    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt" podUID="3c4cedf1-b41a-42b5-9b5f-880b8264d434" containerName="istio-proxy" containerID="cri-o://7125ca9a76e5525938ff5eb853b116962bc21b8375c3319c50ad929221d535d9" gracePeriod=30
Nov 02 02:00:35.674686 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.674662    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 02:00:35.731311 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.731280    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-workload-certs\") pod \"98296768-250b-4160-8678-b92e2fb3a151\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") "
Nov 02 02:00:35.731498 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.731332    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-istio-envoy\") pod \"98296768-250b-4160-8678-b92e2fb3a151\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") "
Nov 02 02:00:35.731498 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.731451    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98296768-250b-4160-8678-b92e2fb3a151-istio-token\" (UniqueName: \"kubernetes.io/projected/98296768-250b-4160-8678-b92e2fb3a151-istio-token\") pod \"98296768-250b-4160-8678-b92e2fb3a151\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") "
Nov 02 02:00:35.731583 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.731523    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98296768-250b-4160-8678-b92e2fb3a151-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98296768-250b-4160-8678-b92e2fb3a151-istio-podinfo\") pod \"98296768-250b-4160-8678-b92e2fb3a151\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") "
Nov 02 02:00:35.731583 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.731547    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-workload-socket\") pod \"98296768-250b-4160-8678-b92e2fb3a151\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") "
Nov 02 02:00:35.731691 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.731600    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-workload-certs" pod "98296768-250b-4160-8678-b92e2fb3a151" (UID: "98296768-250b-4160-8678-b92e2fb3a151"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:35.731757 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.731702    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98296768-250b-4160-8678-b92e2fb3a151-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98296768-250b-4160-8678-b92e2fb3a151-istiod-ca-cert\") pod \"98296768-250b-4160-8678-b92e2fb3a151\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") "
Nov 02 02:00:35.731757 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.731731    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98296768-250b-4160-8678-b92e2fb3a151-kube-api-access-l9c2x\" (UniqueName: \"kubernetes.io/projected/98296768-250b-4160-8678-b92e2fb3a151-kube-api-access-l9c2x\") pod \"98296768-250b-4160-8678-b92e2fb3a151\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") "
Nov 02 02:00:35.731757 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.731749    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-istio-data\") pod \"98296768-250b-4160-8678-b92e2fb3a151\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") "
Nov 02 02:00:35.731923 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.731789    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-credential-socket\") pod \"98296768-250b-4160-8678-b92e2fb3a151\" (UID: \"98296768-250b-4160-8678-b92e2fb3a151\") "
Nov 02 02:00:35.731996 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.731888    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-workload-socket" pod "98296768-250b-4160-8678-b92e2fb3a151" (UID: "98296768-250b-4160-8678-b92e2fb3a151"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:35.732051 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.732035    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-istio-data" pod "98296768-250b-4160-8678-b92e2fb3a151" (UID: "98296768-250b-4160-8678-b92e2fb3a151"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:35.732107 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.732045    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98296768-250b-4160-8678-b92e2fb3a151-istiod-ca-cert" pod "98296768-250b-4160-8678-b92e2fb3a151" (UID: "98296768-250b-4160-8678-b92e2fb3a151"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:35.732399 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.732370    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-credential-socket" pod "98296768-250b-4160-8678-b92e2fb3a151" (UID: "98296768-250b-4160-8678-b92e2fb3a151"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:35.733575 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.733550    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:35.733712 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.733578    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98296768-250b-4160-8678-b92e2fb3a151-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:35.733712 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.733596    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:35.733712 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.733611    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:35.733712 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.733625    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:35.734149 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.734116    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98296768-250b-4160-8678-b92e2fb3a151-istio-token" pod "98296768-250b-4160-8678-b92e2fb3a151" (UID: "98296768-250b-4160-8678-b92e2fb3a151"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:35.734500 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.734477    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98296768-250b-4160-8678-b92e2fb3a151-istio-podinfo" pod "98296768-250b-4160-8678-b92e2fb3a151" (UID: "98296768-250b-4160-8678-b92e2fb3a151"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:00:35.734556 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.734514    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-istio-envoy" pod "98296768-250b-4160-8678-b92e2fb3a151" (UID: "98296768-250b-4160-8678-b92e2fb3a151"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:35.734699 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.734683    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98296768-250b-4160-8678-b92e2fb3a151-kube-api-access-l9c2x" pod "98296768-250b-4160-8678-b92e2fb3a151" (UID: "98296768-250b-4160-8678-b92e2fb3a151"). InnerVolumeSpecName "kube-api-access-l9c2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:35.835642 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.835605    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98296768-250b-4160-8678-b92e2fb3a151-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:35.835642 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.835644    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98296768-250b-4160-8678-b92e2fb3a151-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:35.835912 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.835659    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98296768-250b-4160-8678-b92e2fb3a151-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:35.835912 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:35.835678    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9c2x\" (UniqueName: \"kubernetes.io/projected/98296768-250b-4160-8678-b92e2fb3a151-kube-api-access-l9c2x\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:36.458529 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:36.458498    5875 generic.go:356] "Generic (PLEG): container finished" podID="98296768-250b-4160-8678-b92e2fb3a151" containerID="cd64b3f296c66d4c5bc3bd3c63362e50d740a4b6b8c0e63fd116ee4500f86f80" exitCode=0
Nov 02 02:00:36.458741 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:36.458626    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"
Nov 02 02:00:36.458741 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:36.458645    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz" event={"ID":"98296768-250b-4160-8678-b92e2fb3a151","Type":"ContainerDied","Data":"cd64b3f296c66d4c5bc3bd3c63362e50d740a4b6b8c0e63fd116ee4500f86f80"}
Nov 02 02:00:36.458741 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:36.458703    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz" event={"ID":"98296768-250b-4160-8678-b92e2fb3a151","Type":"ContainerDied","Data":"0927580bdf5b304f3e25a6506fc40b57cd8965e4cfbc7bd7ba84f4a9caa337ce"}
Nov 02 02:00:36.458741 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:36.458724    5875 scope.go:122] "RemoveContainer" containerID="cd64b3f296c66d4c5bc3bd3c63362e50d740a4b6b8c0e63fd116ee4500f86f80"
Nov 02 02:00:36.471586 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:36.471569    5875 scope.go:122] "RemoveContainer" containerID="22a9264bd402ba32e032bfc65f004c848e702ebac53f1a5c3f974db876bf995f"
Nov 02 02:00:36.481509 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:36.481490    5875 scope.go:122] "RemoveContainer" containerID="cd64b3f296c66d4c5bc3bd3c63362e50d740a4b6b8c0e63fd116ee4500f86f80"
Nov 02 02:00:36.481774 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:00:36.481753    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd64b3f296c66d4c5bc3bd3c63362e50d740a4b6b8c0e63fd116ee4500f86f80\": container with ID starting with cd64b3f296c66d4c5bc3bd3c63362e50d740a4b6b8c0e63fd116ee4500f86f80 not found: ID does not exist" containerID="cd64b3f296c66d4c5bc3bd3c63362e50d740a4b6b8c0e63fd116ee4500f86f80"
Nov 02 02:00:36.481841 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:36.481785    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd64b3f296c66d4c5bc3bd3c63362e50d740a4b6b8c0e63fd116ee4500f86f80"} err="failed to get container status \"cd64b3f296c66d4c5bc3bd3c63362e50d740a4b6b8c0e63fd116ee4500f86f80\": rpc error: code = NotFound desc = could not find container \"cd64b3f296c66d4c5bc3bd3c63362e50d740a4b6b8c0e63fd116ee4500f86f80\": container with ID starting with cd64b3f296c66d4c5bc3bd3c63362e50d740a4b6b8c0e63fd116ee4500f86f80 not found: ID does not exist"
Nov 02 02:00:36.481841 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:36.481800    5875 scope.go:122] "RemoveContainer" containerID="22a9264bd402ba32e032bfc65f004c848e702ebac53f1a5c3f974db876bf995f"
Nov 02 02:00:36.482052 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:00:36.482036    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22a9264bd402ba32e032bfc65f004c848e702ebac53f1a5c3f974db876bf995f\": container with ID starting with 22a9264bd402ba32e032bfc65f004c848e702ebac53f1a5c3f974db876bf995f not found: ID does not exist" containerID="22a9264bd402ba32e032bfc65f004c848e702ebac53f1a5c3f974db876bf995f"
Nov 02 02:00:36.482094 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:36.482058    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22a9264bd402ba32e032bfc65f004c848e702ebac53f1a5c3f974db876bf995f"} err="failed to get container status \"22a9264bd402ba32e032bfc65f004c848e702ebac53f1a5c3f974db876bf995f\": rpc error: code = NotFound desc = could not find container \"22a9264bd402ba32e032bfc65f004c848e702ebac53f1a5c3f974db876bf995f\": container with ID starting with 22a9264bd402ba32e032bfc65f004c848e702ebac53f1a5c3f974db876bf995f not found: ID does not exist"
Nov 02 02:00:36.635967 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:36.635931    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"]
Nov 02 02:00:36.642569 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:36.642544    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4f2f926d-openshift-default-5c7c898b87-v6khz"]
Nov 02 02:00:36.719856 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:36.697806    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98296768-250b-4160-8678-b92e2fb3a151" path="/var/lib/kubelet/pods/98296768-250b-4160-8678-b92e2fb3a151/volumes"
Nov 02 02:00:37.623340 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:37.623308    5875 scope.go:122] "RemoveContainer" containerID="aa4da8b0d1745a98b37d1d9abedfcf31f22cc6455e00c7c914f61cce6884d65d"
Nov 02 02:00:37.635046 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:37.635025    5875 scope.go:122] "RemoveContainer" containerID="d83100652a38b0cb03db6caef12b80701adf82cdff8abc5461326841a124ebe5"
Nov 02 02:00:37.645051 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:37.645033    5875 scope.go:122] "RemoveContainer" containerID="252e74770290cf0d7903746086c7c070dedf48d6f53ad4742fcf8a946ac7f75b"
Nov 02 02:00:37.654269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:37.654206    5875 scope.go:122] "RemoveContainer" containerID="94bf7e6eeb574e4ab8d84a06e171599b275d506f03e816ea211bd48490214bfb"
Nov 02 02:00:37.663778 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:37.663755    5875 scope.go:122] "RemoveContainer" containerID="46fd194057cbcf0daae27cabb3539c86733b3ea7f0387642d922fd0df6029f34"
Nov 02 02:00:37.672531 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:37.672516    5875 scope.go:122] "RemoveContainer" containerID="5a42151097f09cbb0759137ebb3074f88589148aa995216c3f33a15819b613ba"
Nov 02 02:00:37.681642 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:37.681622    5875 scope.go:122] "RemoveContainer" containerID="d626d488c15651e4a16ad010ea77207c0b3893bba47132eb478bc9b7ab9c232e"
Nov 02 02:00:37.691655 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:37.691638    5875 scope.go:122] "RemoveContainer" containerID="8c8ba4304a281e3a45fa878c945c97ffc76ca177649fd74435c4226d995b8a7b"
Nov 02 02:00:37.732851 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:37.732831    5875 scope.go:122] "RemoveContainer" containerID="64fd937e8dc09ac685228c3c26dd945b29540dd8a512c86d724a3d494f889cfc"
Nov 02 02:00:37.742898 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:37.742875    5875 scope.go:122] "RemoveContainer" containerID="d6d067f6035cd969943eee705018dd56d746f90fd424b2e5023bda752a4f2323"
Nov 02 02:00:37.752414 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:37.752398    5875 scope.go:122] "RemoveContainer" containerID="8c864f6ca5a4d0ca424c7e94e9f523ed22298ed8c4ab143d4d0a8bb9419120d1"
Nov 02 02:00:40.345499 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.345455    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"]
Nov 02 02:00:40.350190 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.350163    5875 cpu_manager.go:414] "containerMap: removing container" podUID="98296768-250b-4160-8678-b92e2fb3a151" containerName="istio-proxy"
Nov 02 02:00:40.350553 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.350500    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98296768-250b-4160-8678-b92e2fb3a151" containerName="istio-proxy"
Nov 02 02:00:40.358431 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.358406    5875 cpu_manager.go:414] "containerMap: removing container" podUID="8ff0d1e0-810f-4091-92de-6e6575920d13" containerName="cleanup"
Nov 02 02:00:40.358644 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.358626    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ff0d1e0-810f-4091-92de-6e6575920d13" containerName="cleanup"
Nov 02 02:00:40.358765 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.358751    5875 cpu_manager.go:414] "containerMap: removing container" podUID="3fdb61ec-f124-4f24-9ac4-94256e0ab23d" containerName="cleanup"
Nov 02 02:00:40.358883 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.358868    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fdb61ec-f124-4f24-9ac4-94256e0ab23d" containerName="cleanup"
Nov 02 02:00:40.381288 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.376045    5875 cpu_manager.go:414] "containerMap: removing container" podUID="025b1e27-ad55-46ed-a362-ab9e23c4d6d5" containerName="cleanup"
Nov 02 02:00:40.381288 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.376093    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025b1e27-ad55-46ed-a362-ab9e23c4d6d5" containerName="cleanup"
Nov 02 02:00:40.381288 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.376133    5875 cpu_manager.go:414] "containerMap: removing container" podUID="927e4a13-38c2-4b2c-a630-35db116a5ed3" containerName="cleanup"
Nov 02 02:00:40.381288 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.376140    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="927e4a13-38c2-4b2c-a630-35db116a5ed3" containerName="cleanup"
Nov 02 02:00:40.381288 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.376495    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ff0d1e0-810f-4091-92de-6e6575920d13" containerName="cleanup"
Nov 02 02:00:40.381288 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.376516    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fdb61ec-f124-4f24-9ac4-94256e0ab23d" containerName="cleanup"
Nov 02 02:00:40.562319 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.376537    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="025b1e27-ad55-46ed-a362-ab9e23c4d6d5" containerName="cleanup"
Nov 02 02:00:40.562319 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.500339    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="927e4a13-38c2-4b2c-a630-35db116a5ed3" containerName="cleanup"
Nov 02 02:00:40.562319 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.500372    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="98296768-250b-4160-8678-b92e2fb3a151" containerName="istio-proxy"
Nov 02 02:00:40.562319 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.500392    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="98296768-250b-4160-8678-b92e2fb3a151" containerName="istio-proxy"
Nov 02 02:00:40.658405 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.658274    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:40.785244 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.752088    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-8e79d7-openshift-default-dockercfg-hszsw\""
Nov 02 02:00:40.827370 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.787143    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9493769f-3e6b-44f1-aa30-03081e4db485-istio-token\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:40.827370 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.787250    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-credential-socket\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:40.827370 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.787304    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-workload-certs\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:40.827370 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.787344    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h7pr\" (UniqueName: \"kubernetes.io/projected/9493769f-3e6b-44f1-aa30-03081e4db485-kube-api-access-8h7pr\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:40.827370 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.787382    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-istio-envoy\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:40.827370 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.787419    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9493769f-3e6b-44f1-aa30-03081e4db485-istio-podinfo\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:40.827370 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.787605    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-workload-socket\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:40.827370 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.787784    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-istio-data\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:40.827370 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.787868    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9493769f-3e6b-44f1-aa30-03081e4db485-istiod-ca-cert\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:40.955403 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.955366    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9493769f-3e6b-44f1-aa30-03081e4db485-istio-token\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:40.955698 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.955679    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-credential-socket\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:40.955875 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.955857    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-workload-certs\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:40.977120 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.977080    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h7pr\" (UniqueName: \"kubernetes.io/projected/9493769f-3e6b-44f1-aa30-03081e4db485-kube-api-access-8h7pr\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:40.977494 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.977470    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-istio-envoy\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:40.977623 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.977608    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9493769f-3e6b-44f1-aa30-03081e4db485-istio-podinfo\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:40.977871 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.977854    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-workload-socket\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:40.978133 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.978114    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-istio-data\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:40.987261 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:40.978291    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9493769f-3e6b-44f1-aa30-03081e4db485-istiod-ca-cert\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:41.112798 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:41.109898    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-credential-socket\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:41.143584 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:41.121408    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-istio-data\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:41.144115 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:41.144067    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9493769f-3e6b-44f1-aa30-03081e4db485-istiod-ca-cert\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:41.238808 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:41.178586    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-workload-certs\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:41.238808 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:41.183002    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-workload-socket\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:41.238808 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:41.226606    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"]
Nov 02 02:00:41.238808 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:41.227473    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9493769f-3e6b-44f1-aa30-03081e4db485-istio-podinfo\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:41.238808 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:41.228081    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" podUID="6e39fc3d-d730-4df7-8d10-2376656a88d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:00:41.502420 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:41.496476    5875 generic.go:356] "Generic (PLEG): container finished" podID="752ab42e-3d04-4340-8481-9592789c0367" containerID="153ec1dfc61d7fa222cc86a5f48663a9b67b0a75fe80349c483e785922ab015d" exitCode=0
Nov 02 02:00:41.502420 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:41.496637    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz" event={"ID":"752ab42e-3d04-4340-8481-9592789c0367","Type":"ContainerDied","Data":"153ec1dfc61d7fa222cc86a5f48663a9b67b0a75fe80349c483e785922ab015d"}
Nov 02 02:00:41.502420 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:41.497195    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-istio-envoy\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:41.502420 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:41.497882    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9493769f-3e6b-44f1-aa30-03081e4db485-istio-token\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:41.502420 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:41.498480    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h7pr\" (UniqueName: \"kubernetes.io/projected/9493769f-3e6b-44f1-aa30-03081e4db485-kube-api-access-8h7pr\") pod \"e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") " pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:41.619453 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:41.619401    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:00:41.733616 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:41.724982    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:00:42.021336 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.014607    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb" podUID="c9628178-6287-4ba3-9467-93f84ae883ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 02 02:00:42.302769 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.274522    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" podUID="6e39fc3d-d730-4df7-8d10-2376656a88d2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:00:42.302769 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.276128    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 02:00:42.426194 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.426159    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:42.660555 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.530177    5875 generic.go:356] "Generic (PLEG): container finished" podID="0c95a482-2bd1-457e-9f4a-facf829b7699" containerID="d9d62939dac7d40b7165cca55ac30e42ffa615168fcfae9e04fa1bf8c0de3dc6" exitCode=0
Nov 02 02:00:42.660555 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.618368    5875 patch_prober.go:29] interesting pod/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 02:00:42.660555 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.618468    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk" podUID="9e5f896f-d35c-4cbe-87e2-f18aa35b5a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 02:00:42.660555 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.651154    5875 generic.go:356] "Generic (PLEG): container finished" podID="9e5f896f-d35c-4cbe-87e2-f18aa35b5a37" containerID="5fd885597ea0f87801233ad4c73a12efc8774735e406ba072a4100d700fc9106" exitCode=0
Nov 02 02:00:42.660555 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.652156    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g" event={"ID":"0c95a482-2bd1-457e-9f4a-facf829b7699","Type":"ContainerDied","Data":"d9d62939dac7d40b7165cca55ac30e42ffa615168fcfae9e04fa1bf8c0de3dc6"}
Nov 02 02:00:42.660555 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.652239    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk" event={"ID":"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37","Type":"ContainerDied","Data":"5fd885597ea0f87801233ad4c73a12efc8774735e406ba072a4100d700fc9106"}
Nov 02 02:00:42.660555 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.652304    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:00:42.680785 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.680758    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 02:00:42.724126 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.717772    5875 generic.go:356] "Generic (PLEG): container finished" podID="3c4cedf1-b41a-42b5-9b5f-880b8264d434" containerID="7125ca9a76e5525938ff5eb853b116962bc21b8375c3319c50ad929221d535d9" exitCode=0
Nov 02 02:00:42.735756 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:00:42.735724    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:00:42.737092 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.737067    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 02:00:42.771449 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.770001    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt" event={"ID":"3c4cedf1-b41a-42b5-9b5f-880b8264d434","Type":"ContainerDied","Data":"7125ca9a76e5525938ff5eb853b116962bc21b8375c3319c50ad929221d535d9"}
Nov 02 02:00:42.771449 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.770071    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 02:00:42.771449 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.770087    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-af807eae-tpnmf"]
Nov 02 02:00:42.771449 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.770104    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-af807eae-tpnmf"]
Nov 02 02:00:42.771449 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.770120    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"]
Nov 02 02:00:42.771449 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.770154    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 02:00:42.771449 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.770169    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"]
Nov 02 02:00:42.771449 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.770189    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"]
Nov 02 02:00:42.771449 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.770206    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"]
Nov 02 02:00:42.771449 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.770244    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"]
Nov 02 02:00:42.771449 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.770424    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj" podUID="be86e257-8a07-418f-a12a-339e09651c93" containerName="istio-proxy" containerID="cri-o://3a8c2fd55e1d3b8c9b92e4c787133742599a5d43de66e20ab3e5c1c73d532d1c" gracePeriod=30
Nov 02 02:00:42.771449 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.770630    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45" podUID="77d448cc-8da2-4cbc-9d7e-240e51a096c0" containerName="istio-proxy" containerID="cri-o://896a02bf2e1970079559dba3d178c18f3bdab2c34502fc0c79d37e0fe13f0be5" gracePeriod=30
Nov 02 02:00:42.771449 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.770824    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw" podUID="166343e8-524c-4668-9094-a503991be4f9" containerName="istio-proxy" containerID="cri-o://e363896987f9192d328bf8d611238d41fa218712e4fa3794af11dbb506c6843a" gracePeriod=30
Nov 02 02:00:42.771449 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.771007    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph" podUID="fb26c061-10c7-4d0c-896c-1fbd27ed00aa" containerName="istio-proxy" containerID="cri-o://3e906e3ea143f21d04aa923cab25c201ce66e79724cc807bab6c10f3a876d9af" gracePeriod=30
Nov 02 02:00:42.771449 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.771308    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb" podUID="ff6cb777-acc6-4c58-8302-26d6e2906cdf" containerName="istio-proxy" containerID="cri-o://97de7d64ce55692ccda83e9d5866414b1d18cf9d5fc60159179055c4e5e2207a" gracePeriod=30
Nov 02 02:00:42.778411 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.778385    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 02:00:42.782132 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.782047    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 02:00:42.800729 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.800696    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-token\" (UniqueName: \"kubernetes.io/projected/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-token\") pod \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") "
Nov 02 02:00:42.802874 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.801202    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-data\") pod \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") "
Nov 02 02:00:42.802874 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.801299    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-envoy\") pod \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") "
Nov 02 02:00:42.802874 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.801367    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-workload-certs\") pod \"0c95a482-2bd1-457e-9f4a-facf829b7699\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") "
Nov 02 02:00:42.802874 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.801407    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-workload-certs\") pod \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") "
Nov 02 02:00:42.802874 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.801438    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-credential-socket\") pod \"0c95a482-2bd1-457e-9f4a-facf829b7699\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") "
Nov 02 02:00:42.803541 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.803253    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-data" pod "3c4cedf1-b41a-42b5-9b5f-880b8264d434" (UID: "3c4cedf1-b41a-42b5-9b5f-880b8264d434"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:42.804241 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.804057    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-workload-certs" pod "0c95a482-2bd1-457e-9f4a-facf829b7699" (UID: "0c95a482-2bd1-457e-9f4a-facf829b7699"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:42.804600 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.804398    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-workload-certs" pod "3c4cedf1-b41a-42b5-9b5f-880b8264d434" (UID: "3c4cedf1-b41a-42b5-9b5f-880b8264d434"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:42.805246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.804740    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-credential-socket" pod "0c95a482-2bd1-457e-9f4a-facf829b7699" (UID: "0c95a482-2bd1-457e-9f4a-facf829b7699"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:42.806737 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.806701    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-envoy" pod "3c4cedf1-b41a-42b5-9b5f-880b8264d434" (UID: "3c4cedf1-b41a-42b5-9b5f-880b8264d434"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:42.807057 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.807021    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-token" pod "3c4cedf1-b41a-42b5-9b5f-880b8264d434" (UID: "3c4cedf1-b41a-42b5-9b5f-880b8264d434"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:42.823126 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.823104    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 02:00:42.903083 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.903004    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-envoy\") pod \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") "
Nov 02 02:00:42.903083 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.903057    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-workload-socket\") pod \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") "
Nov 02 02:00:42.903379 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.903091    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-istio-data\") pod \"752ab42e-3d04-4340-8481-9592789c0367\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") "
Nov 02 02:00:42.903379 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.903151    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-podinfo\") pod \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") "
Nov 02 02:00:42.903379 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.903192    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istiod-ca-cert\") pod \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") "
Nov 02 02:00:42.903379 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.903285    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-token\" (UniqueName: \"kubernetes.io/projected/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-token\") pod \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") "
Nov 02 02:00:42.903379 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.903361    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-workload-socket" pod "9e5f896f-d35c-4cbe-87e2-f18aa35b5a37" (UID: "9e5f896f-d35c-4cbe-87e2-f18aa35b5a37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:42.903658 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.903380    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/752ab42e-3d04-4340-8481-9592789c0367-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/752ab42e-3d04-4340-8481-9592789c0367-istiod-ca-cert\") pod \"752ab42e-3d04-4340-8481-9592789c0367\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") "
Nov 02 02:00:42.903658 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.903417    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istiod-ca-cert\") pod \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") "
Nov 02 02:00:42.903658 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.903450    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/752ab42e-3d04-4340-8481-9592789c0367-kube-api-access-v5482\" (UniqueName: \"kubernetes.io/projected/752ab42e-3d04-4340-8481-9592789c0367-kube-api-access-v5482\") pod \"752ab42e-3d04-4340-8481-9592789c0367\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") "
Nov 02 02:00:42.903658 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.903502    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-data\") pod \"0c95a482-2bd1-457e-9f4a-facf829b7699\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") "
Nov 02 02:00:42.903658 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.903541    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/752ab42e-3d04-4340-8481-9592789c0367-istio-token\" (UniqueName: \"kubernetes.io/projected/752ab42e-3d04-4340-8481-9592789c0367-istio-token\") pod \"752ab42e-3d04-4340-8481-9592789c0367\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") "
Nov 02 02:00:42.903658 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.903576    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-kube-api-access-lxd2h\" (UniqueName: \"kubernetes.io/projected/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-kube-api-access-lxd2h\") pod \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") "
Nov 02 02:00:42.903658 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.903607    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-envoy\") pod \"0c95a482-2bd1-457e-9f4a-facf829b7699\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") "
Nov 02 02:00:42.904330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.903675    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-podinfo\") pod \"0c95a482-2bd1-457e-9f4a-facf829b7699\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") "
Nov 02 02:00:42.904330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.903706    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-workload-certs\") pod \"752ab42e-3d04-4340-8481-9592789c0367\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") "
Nov 02 02:00:42.904330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.903740    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-workload-socket\") pod \"0c95a482-2bd1-457e-9f4a-facf829b7699\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") "
Nov 02 02:00:42.904330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.903776    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-credential-socket\") pod \"752ab42e-3d04-4340-8481-9592789c0367\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") "
Nov 02 02:00:42.904330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.903835    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-credential-socket\") pod \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") "
Nov 02 02:00:42.904330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.903863    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-token\" (UniqueName: \"kubernetes.io/projected/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-token\") pod \"0c95a482-2bd1-457e-9f4a-facf829b7699\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") "
Nov 02 02:00:42.904330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.903938    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-credential-socket\") pod \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") "
Nov 02 02:00:42.904330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.903972    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-workload-socket\") pod \"752ab42e-3d04-4340-8481-9592789c0367\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") "
Nov 02 02:00:42.904330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.904005    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c95a482-2bd1-457e-9f4a-facf829b7699-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c95a482-2bd1-457e-9f4a-facf829b7699-istiod-ca-cert\") pod \"0c95a482-2bd1-457e-9f4a-facf829b7699\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") "
Nov 02 02:00:42.904330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.904078    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-podinfo\") pod \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") "
Nov 02 02:00:42.904330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.904109    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-istio-envoy\") pod \"752ab42e-3d04-4340-8481-9592789c0367\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") "
Nov 02 02:00:42.904330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.904151    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/752ab42e-3d04-4340-8481-9592789c0367-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/752ab42e-3d04-4340-8481-9592789c0367-istio-podinfo\") pod \"752ab42e-3d04-4340-8481-9592789c0367\" (UID: \"752ab42e-3d04-4340-8481-9592789c0367\") "
Nov 02 02:00:42.904330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.904289    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-workload-certs\") pod \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") "
Nov 02 02:00:42.905111 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.904369    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-data\") pod \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\" (UID: \"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37\") "
Nov 02 02:00:42.905111 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.904422    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-workload-socket\") pod \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") "
Nov 02 02:00:42.905111 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.904472    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c95a482-2bd1-457e-9f4a-facf829b7699-kube-api-access-7lnvd\" (UniqueName: \"kubernetes.io/projected/0c95a482-2bd1-457e-9f4a-facf829b7699-kube-api-access-7lnvd\") pod \"0c95a482-2bd1-457e-9f4a-facf829b7699\" (UID: \"0c95a482-2bd1-457e-9f4a-facf829b7699\") "
Nov 02 02:00:42.905111 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.904536    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c4cedf1-b41a-42b5-9b5f-880b8264d434-kube-api-access-jpg5x\" (UniqueName: \"kubernetes.io/projected/3c4cedf1-b41a-42b5-9b5f-880b8264d434-kube-api-access-jpg5x\") pod \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\" (UID: \"3c4cedf1-b41a-42b5-9b5f-880b8264d434\") "
Nov 02 02:00:42.906983 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.905580    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:42.906983 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.905608    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:42.906983 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.905623    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:42.906983 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.905639    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:42.906983 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.905652    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:42.906983 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.905667    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:42.906983 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.905681    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:42.906983 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.905885    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-credential-socket" pod "9e5f896f-d35c-4cbe-87e2-f18aa35b5a37" (UID: "9e5f896f-d35c-4cbe-87e2-f18aa35b5a37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:42.906983 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.906191    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-istio-data" pod "752ab42e-3d04-4340-8481-9592789c0367" (UID: "752ab42e-3d04-4340-8481-9592789c0367"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:42.906983 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.906098    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istiod-ca-cert" pod "9e5f896f-d35c-4cbe-87e2-f18aa35b5a37" (UID: "9e5f896f-d35c-4cbe-87e2-f18aa35b5a37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:42.906983 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.906349    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/752ab42e-3d04-4340-8481-9592789c0367-istiod-ca-cert" pod "752ab42e-3d04-4340-8481-9592789c0367" (UID: "752ab42e-3d04-4340-8481-9592789c0367"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:42.906983 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.906677    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istiod-ca-cert" pod "3c4cedf1-b41a-42b5-9b5f-880b8264d434" (UID: "3c4cedf1-b41a-42b5-9b5f-880b8264d434"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:42.906983 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.906779    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-credential-socket" pod "3c4cedf1-b41a-42b5-9b5f-880b8264d434" (UID: "3c4cedf1-b41a-42b5-9b5f-880b8264d434"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:42.907674 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.906986    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-workload-socket" pod "752ab42e-3d04-4340-8481-9592789c0367" (UID: "752ab42e-3d04-4340-8481-9592789c0367"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:42.907674 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.907304    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c95a482-2bd1-457e-9f4a-facf829b7699-istiod-ca-cert" pod "0c95a482-2bd1-457e-9f4a-facf829b7699" (UID: "0c95a482-2bd1-457e-9f4a-facf829b7699"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:42.909063 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.908700    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-workload-certs" pod "752ab42e-3d04-4340-8481-9592789c0367" (UID: "752ab42e-3d04-4340-8481-9592789c0367"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:42.909063 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.908708    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-workload-socket" pod "3c4cedf1-b41a-42b5-9b5f-880b8264d434" (UID: "3c4cedf1-b41a-42b5-9b5f-880b8264d434"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:42.909063 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.908977    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-workload-socket" pod "0c95a482-2bd1-457e-9f4a-facf829b7699" (UID: "0c95a482-2bd1-457e-9f4a-facf829b7699"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:42.909303 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.909109    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-workload-certs" pod "9e5f896f-d35c-4cbe-87e2-f18aa35b5a37" (UID: "9e5f896f-d35c-4cbe-87e2-f18aa35b5a37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:42.909303 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.909134    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-data" pod "0c95a482-2bd1-457e-9f4a-facf829b7699" (UID: "0c95a482-2bd1-457e-9f4a-facf829b7699"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:42.909479 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.909455    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-credential-socket" pod "752ab42e-3d04-4340-8481-9592789c0367" (UID: "752ab42e-3d04-4340-8481-9592789c0367"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:42.909542 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.909502    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-data" pod "9e5f896f-d35c-4cbe-87e2-f18aa35b5a37" (UID: "9e5f896f-d35c-4cbe-87e2-f18aa35b5a37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:42.911016 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.910859    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-envoy" pod "9e5f896f-d35c-4cbe-87e2-f18aa35b5a37" (UID: "9e5f896f-d35c-4cbe-87e2-f18aa35b5a37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:42.911016 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.910987    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-token" pod "0c95a482-2bd1-457e-9f4a-facf829b7699" (UID: "0c95a482-2bd1-457e-9f4a-facf829b7699"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:42.912585 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.912420    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-token" pod "9e5f896f-d35c-4cbe-87e2-f18aa35b5a37" (UID: "9e5f896f-d35c-4cbe-87e2-f18aa35b5a37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:42.912585 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.912482    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-podinfo" pod "9e5f896f-d35c-4cbe-87e2-f18aa35b5a37" (UID: "9e5f896f-d35c-4cbe-87e2-f18aa35b5a37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:00:42.912585 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.912561    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-podinfo" pod "3c4cedf1-b41a-42b5-9b5f-880b8264d434" (UID: "3c4cedf1-b41a-42b5-9b5f-880b8264d434"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:00:42.913568 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.913419    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-istio-envoy" pod "752ab42e-3d04-4340-8481-9592789c0367" (UID: "752ab42e-3d04-4340-8481-9592789c0367"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:42.913568 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.913434    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/752ab42e-3d04-4340-8481-9592789c0367-istio-podinfo" pod "752ab42e-3d04-4340-8481-9592789c0367" (UID: "752ab42e-3d04-4340-8481-9592789c0367"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:00:42.913568 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.913515    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4cedf1-b41a-42b5-9b5f-880b8264d434-kube-api-access-jpg5x" pod "3c4cedf1-b41a-42b5-9b5f-880b8264d434" (UID: "3c4cedf1-b41a-42b5-9b5f-880b8264d434"). InnerVolumeSpecName "kube-api-access-jpg5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:42.913568 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.913540    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-kube-api-access-lxd2h" pod "9e5f896f-d35c-4cbe-87e2-f18aa35b5a37" (UID: "9e5f896f-d35c-4cbe-87e2-f18aa35b5a37"). InnerVolumeSpecName "kube-api-access-lxd2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:42.913994 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.913957    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752ab42e-3d04-4340-8481-9592789c0367-kube-api-access-v5482" pod "752ab42e-3d04-4340-8481-9592789c0367" (UID: "752ab42e-3d04-4340-8481-9592789c0367"). InnerVolumeSpecName "kube-api-access-v5482". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:42.915266 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.915197    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c95a482-2bd1-457e-9f4a-facf829b7699-kube-api-access-7lnvd" pod "0c95a482-2bd1-457e-9f4a-facf829b7699" (UID: "0c95a482-2bd1-457e-9f4a-facf829b7699"). InnerVolumeSpecName "kube-api-access-7lnvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:42.915398 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.915376    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-podinfo" pod "0c95a482-2bd1-457e-9f4a-facf829b7699" (UID: "0c95a482-2bd1-457e-9f4a-facf829b7699"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:00:42.915473 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.915437    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752ab42e-3d04-4340-8481-9592789c0367-istio-token" pod "752ab42e-3d04-4340-8481-9592789c0367" (UID: "752ab42e-3d04-4340-8481-9592789c0367"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:42.915517 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:42.915499    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-envoy" pod "0c95a482-2bd1-457e-9f4a-facf829b7699" (UID: "0c95a482-2bd1-457e-9f4a-facf829b7699"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:43.007664 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.007625    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.007664 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.007665    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.007999 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.007680    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/752ab42e-3d04-4340-8481-9592789c0367-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.007999 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.007693    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.007999 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.007833    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5482\" (UniqueName: \"kubernetes.io/projected/752ab42e-3d04-4340-8481-9592789c0367-kube-api-access-v5482\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.007999 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.007856    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.007999 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.007874    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/752ab42e-3d04-4340-8481-9592789c0367-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.007999 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.007888    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxd2h\" (UniqueName: \"kubernetes.io/projected/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-kube-api-access-lxd2h\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.007999 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.007902    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.007999 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.007916    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.007999 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.007929    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.007999 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.007944    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c95a482-2bd1-457e-9f4a-facf829b7699-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.007999 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.007960    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.007999 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.007976    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.007999 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.007990    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c95a482-2bd1-457e-9f4a-facf829b7699-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.007999 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.008004    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.007999 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.008017    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.009673 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.008034    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c95a482-2bd1-457e-9f4a-facf829b7699-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.009673 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.008055    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4cedf1-b41a-42b5-9b5f-880b8264d434-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.009673 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.008069    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.009673 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.008081    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/752ab42e-3d04-4340-8481-9592789c0367-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.009673 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.008097    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.009673 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.008116    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.009673 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.008128    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4cedf1-b41a-42b5-9b5f-880b8264d434-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.009673 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.008143    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lnvd\" (UniqueName: \"kubernetes.io/projected/0c95a482-2bd1-457e-9f4a-facf829b7699-kube-api-access-7lnvd\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.009673 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.008159    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpg5x\" (UniqueName: \"kubernetes.io/projected/3c4cedf1-b41a-42b5-9b5f-880b8264d434-kube-api-access-jpg5x\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.009673 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.008173    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.009673 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.008187    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/752ab42e-3d04-4340-8481-9592789c0367-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.009673 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.008199    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:43.225042 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:00:43.225014    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9493769f_3e6b_44f1_aa30_03081e4db485.slice/crio-8cd688c8c8a26594574a8f3727416d9b753f41d3d85f0f099357e60f173f50f1 WatchSource:0}: Error finding container 8cd688c8c8a26594574a8f3727416d9b753f41d3d85f0f099357e60f173f50f1: Status 404 returned error can't find the container with id 8cd688c8c8a26594574a8f3727416d9b753f41d3d85f0f099357e60f173f50f1
Nov 02 02:00:43.225424 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.225399    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"]
Nov 02 02:00:43.227838 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.227804    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:00:43.227939 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.227879    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:00:43.227939 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.227925    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:00:43.806316 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.802911    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt" event={"ID":"3c4cedf1-b41a-42b5-9b5f-880b8264d434","Type":"ContainerDied","Data":"93406a1e36afed6c0e45b7e2d96335ca9a17d8a62511281f2f0ddb0dcd4f2ff2"}
Nov 02 02:00:43.806316 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.802991    5875 scope.go:122] "RemoveContainer" containerID="7125ca9a76e5525938ff5eb853b116962bc21b8375c3319c50ad929221d535d9"
Nov 02 02:00:43.806316 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.803145    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"
Nov 02 02:00:43.814566 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.810089    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz" event={"ID":"752ab42e-3d04-4340-8481-9592789c0367","Type":"ContainerDied","Data":"6ce74dfffff3d4f93804f1e187ab9b835216267e410e50342f93505ce8ee1290"}
Nov 02 02:00:43.814566 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.810203    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"
Nov 02 02:00:43.822109 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.820431    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g" event={"ID":"0c95a482-2bd1-457e-9f4a-facf829b7699","Type":"ContainerDied","Data":"a8b2251a114bc3be50961c406e5b14d1ef6c2b30d14df6521638a0db962a3e05"}
Nov 02 02:00:43.822109 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.820563    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"
Nov 02 02:00:43.828529 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.828503    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7" event={"ID":"9493769f-3e6b-44f1-aa30-03081e4db485","Type":"ContainerStarted","Data":"581f5dfbafadadcab3095717730bee01dfc9eed93b78c8791fed28e4dc574558"}
Nov 02 02:00:43.828624 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.828553    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7" event={"ID":"9493769f-3e6b-44f1-aa30-03081e4db485","Type":"ContainerStarted","Data":"8cd688c8c8a26594574a8f3727416d9b753f41d3d85f0f099357e60f173f50f1"}
Nov 02 02:00:43.835384 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.835362    5875 scope.go:122] "RemoveContainer" containerID="153ec1dfc61d7fa222cc86a5f48663a9b67b0a75fe80349c483e785922ab015d"
Nov 02 02:00:43.836591 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.836574    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"
Nov 02 02:00:43.836875 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.836852    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk" event={"ID":"9e5f896f-d35c-4cbe-87e2-f18aa35b5a37","Type":"ContainerDied","Data":"97514ae56fba0b2933fcb454f1c953b3cb05d14ce67c1ef428168fd2a19db268"}
Nov 02 02:00:43.854191 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.854165    5875 scope.go:122] "RemoveContainer" containerID="d9d62939dac7d40b7165cca55ac30e42ffa615168fcfae9e04fa1bf8c0de3dc6"
Nov 02 02:00:43.896730 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:43.896709    5875 scope.go:122] "RemoveContainer" containerID="5fd885597ea0f87801233ad4c73a12efc8774735e406ba072a4100d700fc9106"
Nov 02 02:00:44.299756 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.299721    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"]
Nov 02 02:00:44.303917 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.303893    5875 cpu_manager.go:414] "containerMap: removing container" podUID="9e5f896f-d35c-4cbe-87e2-f18aa35b5a37" containerName="istio-proxy"
Nov 02 02:00:44.303917 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.303918    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e5f896f-d35c-4cbe-87e2-f18aa35b5a37" containerName="istio-proxy"
Nov 02 02:00:44.304072 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.303970    5875 cpu_manager.go:414] "containerMap: removing container" podUID="752ab42e-3d04-4340-8481-9592789c0367" containerName="istio-proxy"
Nov 02 02:00:44.304072 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.303978    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="752ab42e-3d04-4340-8481-9592789c0367" containerName="istio-proxy"
Nov 02 02:00:44.304072 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.304017    5875 cpu_manager.go:414] "containerMap: removing container" podUID="98296768-250b-4160-8678-b92e2fb3a151" containerName="istio-proxy"
Nov 02 02:00:44.304072 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.304024    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98296768-250b-4160-8678-b92e2fb3a151" containerName="istio-proxy"
Nov 02 02:00:44.304072 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.304037    5875 cpu_manager.go:414] "containerMap: removing container" podUID="0c95a482-2bd1-457e-9f4a-facf829b7699" containerName="istio-proxy"
Nov 02 02:00:44.304072 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.304046    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c95a482-2bd1-457e-9f4a-facf829b7699" containerName="istio-proxy"
Nov 02 02:00:44.304072 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.304072    5875 cpu_manager.go:414] "containerMap: removing container" podUID="3c4cedf1-b41a-42b5-9b5f-880b8264d434" containerName="istio-proxy"
Nov 02 02:00:44.304358 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.304078    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c4cedf1-b41a-42b5-9b5f-880b8264d434" containerName="istio-proxy"
Nov 02 02:00:44.304358 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.304331    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e5f896f-d35c-4cbe-87e2-f18aa35b5a37" containerName="istio-proxy"
Nov 02 02:00:44.304358 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.304354    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="752ab42e-3d04-4340-8481-9592789c0367" containerName="istio-proxy"
Nov 02 02:00:44.304451 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.304369    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c95a482-2bd1-457e-9f4a-facf829b7699" containerName="istio-proxy"
Nov 02 02:00:44.304451 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.304382    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c4cedf1-b41a-42b5-9b5f-880b8264d434" containerName="istio-proxy"
Nov 02 02:00:44.314657 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.314633    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.343597 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.343565    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"]
Nov 02 02:00:44.355878 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.355852    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-bbrhk"]
Nov 02 02:00:44.365364 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.365338    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"]
Nov 02 02:00:44.367398 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.367374    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-c1aad5-openshift-default-dockercfg-8lcpp\""
Nov 02 02:00:44.370619 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.370594    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-envoy\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.370729 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.370636    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-podinfo\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.370729 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.370675    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istiod-ca-cert\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.370863 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.370845    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-workload-socket\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.370957 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.370927    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-credential-socket\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.371024 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.371012    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-token\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.371087 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.371062    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52w5b\" (UniqueName: \"kubernetes.io/projected/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-kube-api-access-52w5b\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.371143 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.371106    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-workload-certs\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.371196 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.371170    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-data\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.500248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.489717    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-envoy\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.500248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.489786    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-podinfo\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.500248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.489830    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istiod-ca-cert\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.500248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.490055    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-workload-socket\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.500248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.490173    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-credential-socket\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.500248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.490272    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-token\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.500248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.490334    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52w5b\" (UniqueName: \"kubernetes.io/projected/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-kube-api-access-52w5b\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.500248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.490397    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-workload-certs\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.500248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.490479    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-data\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.500248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.496948    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-data\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.522245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.515280    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-envoy\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.522245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.515358    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"]
Nov 02 02:00:44.522245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.516906    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-credential-socket\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.522245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.520487    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-podinfo\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.522245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.521027    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-workload-certs\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.522245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.521267    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istiod-ca-cert\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.522245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.521490    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-workload-socket\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.556749 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.556595    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-fqmqt"]
Nov 02 02:00:44.615330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.587640    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"]
Nov 02 02:00:44.615330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.595997    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:44.615330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.603502    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-9xg5g"]
Nov 02 02:00:44.615330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.607421    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-token\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.619065 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.618999    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52w5b\" (UniqueName: \"kubernetes.io/projected/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-kube-api-access-52w5b\") pod \"e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") " pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.625589 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.625521    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:44.655646 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.655600    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:44.699849 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.699822    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c95a482-2bd1-457e-9f4a-facf829b7699" path="/var/lib/kubelet/pods/0c95a482-2bd1-457e-9f4a-facf829b7699/volumes"
Nov 02 02:00:44.700641 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.700621    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f20d922-b401-4f5d-800b-64db3a04af32" path="/var/lib/kubelet/pods/2f20d922-b401-4f5d-800b-64db3a04af32/volumes"
Nov 02 02:00:44.701250 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.701202    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c4cedf1-b41a-42b5-9b5f-880b8264d434" path="/var/lib/kubelet/pods/3c4cedf1-b41a-42b5-9b5f-880b8264d434/volumes"
Nov 02 02:00:44.702198 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.702181    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e5f896f-d35c-4cbe-87e2-f18aa35b5a37" path="/var/lib/kubelet/pods/9e5f896f-d35c-4cbe-87e2-f18aa35b5a37/volumes"
Nov 02 02:00:44.703051 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.703036    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"]
Nov 02 02:00:44.710547 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.710526    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9470a3-openshift-default-6769c45f77-dcgwz"]
Nov 02 02:00:44.844512 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.844408    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7" podStartSLOduration=4.844389233 podStartE2EDuration="4.844389233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:44.839978873 +0000 UTC m=+1692.281015370" watchObservedRunningTime="2026-11-02 02:00:44.844389233 +0000 UTC m=+1692.285425731"
Nov 02 02:00:44.860916 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.860653    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:44.861637 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:44.861618    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:00:45.044497 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:45.044107    5875 patch_prober.go:29] interesting pod/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:00:45.044497 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:45.044178    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg" podUID="9a054e97-d2a9-47fd-a380-24f4c97d908e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:00:45.264607 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:45.263522    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"]
Nov 02 02:00:45.275109 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:45.275069    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:00:45.275237 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:45.275175    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:00:45.275303 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:45.275243    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:00:45.871532 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:45.871490    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg" event={"ID":"d1dafba9-1caf-4355-bfa1-a3b8cea9155e","Type":"ContainerStarted","Data":"a0089823b223237cf52b7b36d05236b80bdae136acb09f33349f664343d476a6"}
Nov 02 02:00:45.872002 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:45.871592    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg" event={"ID":"d1dafba9-1caf-4355-bfa1-a3b8cea9155e","Type":"ContainerStarted","Data":"f3f445947fd406fa4d56e6ccff3588b44bf1f96bcc4a8b3a52728cabca976499"}
Nov 02 02:00:46.000770 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:46.000697    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg" podStartSLOduration=2.000679364 podStartE2EDuration="2.000679364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:45.995035542 +0000 UTC m=+1693.436072039" watchObservedRunningTime="2026-11-02 02:00:46.000679364 +0000 UTC m=+1693.441715862"
Nov 02 02:00:46.250041 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:46.249685    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:00:46.250041 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:46.249757    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph" podUID="fb26c061-10c7-4d0c-896c-1fbd27ed00aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:00:46.656150 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:46.656076    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:46.661258 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:46.661198    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:46.697794 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:46.697762    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="752ab42e-3d04-4340-8481-9592789c0367" path="/var/lib/kubelet/pods/752ab42e-3d04-4340-8481-9592789c0367/volumes"
Nov 02 02:00:46.936306 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:46.935914    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:46.956248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:46.952582    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:00:47.970307 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:47.969936    5875 generic.go:356] "Generic (PLEG): container finished" podID="ff6cb777-acc6-4c58-8302-26d6e2906cdf" containerID="97de7d64ce55692ccda83e9d5866414b1d18cf9d5fc60159179055c4e5e2207a" exitCode=0
Nov 02 02:00:47.970307 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:47.970093    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb" event={"ID":"ff6cb777-acc6-4c58-8302-26d6e2906cdf","Type":"ContainerDied","Data":"97de7d64ce55692ccda83e9d5866414b1d18cf9d5fc60159179055c4e5e2207a"}
Nov 02 02:00:47.970307 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:47.970172    5875 scope.go:122] "RemoveContainer" containerID="1af18b7499f5362c05ea1b91c9d3b741f1bf47f262cd701f6a6b9e10e0c3c86d"
Nov 02 02:00:47.974180 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:47.974152    5875 generic.go:356] "Generic (PLEG): container finished" podID="166343e8-524c-4668-9094-a503991be4f9" containerID="e363896987f9192d328bf8d611238d41fa218712e4fa3794af11dbb506c6843a" exitCode=0
Nov 02 02:00:47.975542 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:47.975409    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw" event={"ID":"166343e8-524c-4668-9094-a503991be4f9","Type":"ContainerDied","Data":"e363896987f9192d328bf8d611238d41fa218712e4fa3794af11dbb506c6843a"}
Nov 02 02:00:48.051353 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.051320    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 02:00:48.051496 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.051405    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj" podUID="be86e257-8a07-418f-a12a-339e09651c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 02:00:48.095912 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.095857    5875 scope.go:122] "RemoveContainer" containerID="8f3f00f580c3ccd7ee044a5da972c67d9186da7c6a00b8cb8c07d52dc9b8b51d"
Nov 02 02:00:48.207033 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.206995    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 02:00:48.218122 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.218091    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 02:00:48.236993 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.236959    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-istio-data\") pod \"166343e8-524c-4668-9094-a503991be4f9\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") "
Nov 02 02:00:48.237136 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.237010    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-token\" (UniqueName: \"kubernetes.io/projected/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-token\") pod \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") "
Nov 02 02:00:48.237136 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.237049    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-kube-api-access-bkqjq\" (UniqueName: \"kubernetes.io/projected/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-kube-api-access-bkqjq\") pod \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") "
Nov 02 02:00:48.237136 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.237070    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-envoy\") pod \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") "
Nov 02 02:00:48.237136 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.237118    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-podinfo\") pod \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") "
Nov 02 02:00:48.237324 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.237185    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-credential-socket\") pod \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") "
Nov 02 02:00:48.237324 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.237214    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-workload-certs\") pod \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") "
Nov 02 02:00:48.237324 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.237313    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-credential-socket\") pod \"166343e8-524c-4668-9094-a503991be4f9\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") "
Nov 02 02:00:48.237441 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.237350    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/166343e8-524c-4668-9094-a503991be4f9-istio-token\" (UniqueName: \"kubernetes.io/projected/166343e8-524c-4668-9094-a503991be4f9-istio-token\") pod \"166343e8-524c-4668-9094-a503991be4f9\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") "
Nov 02 02:00:48.237499 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.237480    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/166343e8-524c-4668-9094-a503991be4f9-kube-api-access-4jw7k\" (UniqueName: \"kubernetes.io/projected/166343e8-524c-4668-9094-a503991be4f9-kube-api-access-4jw7k\") pod \"166343e8-524c-4668-9094-a503991be4f9\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") "
Nov 02 02:00:48.237536 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.237516    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-istio-data" pod "166343e8-524c-4668-9094-a503991be4f9" (UID: "166343e8-524c-4668-9094-a503991be4f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:48.237596 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.237545    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istiod-ca-cert\") pod \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") "
Nov 02 02:00:48.237641 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.237612    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-workload-certs\") pod \"166343e8-524c-4668-9094-a503991be4f9\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") "
Nov 02 02:00:48.237682 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.237644    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-workload-socket\") pod \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") "
Nov 02 02:00:48.237682 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.237666    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-workload-socket\") pod \"166343e8-524c-4668-9094-a503991be4f9\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") "
Nov 02 02:00:48.237781 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.237696    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-istio-envoy\") pod \"166343e8-524c-4668-9094-a503991be4f9\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") "
Nov 02 02:00:48.237781 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.237729    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-data\") pod \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\" (UID: \"fb26c061-10c7-4d0c-896c-1fbd27ed00aa\") "
Nov 02 02:00:48.237781 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.237764    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/166343e8-524c-4668-9094-a503991be4f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/166343e8-524c-4668-9094-a503991be4f9-istiod-ca-cert\") pod \"166343e8-524c-4668-9094-a503991be4f9\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") "
Nov 02 02:00:48.237781 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.237770    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-credential-socket" pod "fb26c061-10c7-4d0c-896c-1fbd27ed00aa" (UID: "fb26c061-10c7-4d0c-896c-1fbd27ed00aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:48.237989 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.237785    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/166343e8-524c-4668-9094-a503991be4f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/166343e8-524c-4668-9094-a503991be4f9-istio-podinfo\") pod \"166343e8-524c-4668-9094-a503991be4f9\" (UID: \"166343e8-524c-4668-9094-a503991be4f9\") "
Nov 02 02:00:48.238497 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.238294    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-workload-certs" pod "166343e8-524c-4668-9094-a503991be4f9" (UID: "166343e8-524c-4668-9094-a503991be4f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:48.238612 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.238501    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.238612 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.238522    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.238612 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.238540    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.238779 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.238740    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-data" pod "fb26c061-10c7-4d0c-896c-1fbd27ed00aa" (UID: "fb26c061-10c7-4d0c-896c-1fbd27ed00aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:48.239327 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.238998    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istiod-ca-cert" pod "fb26c061-10c7-4d0c-896c-1fbd27ed00aa" (UID: "fb26c061-10c7-4d0c-896c-1fbd27ed00aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:48.239327 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.239023    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-workload-certs" pod "fb26c061-10c7-4d0c-896c-1fbd27ed00aa" (UID: "fb26c061-10c7-4d0c-896c-1fbd27ed00aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:48.239327 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.239031    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/166343e8-524c-4668-9094-a503991be4f9-istiod-ca-cert" pod "166343e8-524c-4668-9094-a503991be4f9" (UID: "166343e8-524c-4668-9094-a503991be4f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:48.239327 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.239317    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-credential-socket" pod "166343e8-524c-4668-9094-a503991be4f9" (UID: "166343e8-524c-4668-9094-a503991be4f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:48.239592 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.239317    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-workload-socket" pod "fb26c061-10c7-4d0c-896c-1fbd27ed00aa" (UID: "fb26c061-10c7-4d0c-896c-1fbd27ed00aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:48.239924 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.239664    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-workload-socket" pod "166343e8-524c-4668-9094-a503991be4f9" (UID: "166343e8-524c-4668-9094-a503991be4f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:48.242584 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.242557    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/166343e8-524c-4668-9094-a503991be4f9-kube-api-access-4jw7k" pod "166343e8-524c-4668-9094-a503991be4f9" (UID: "166343e8-524c-4668-9094-a503991be4f9"). InnerVolumeSpecName "kube-api-access-4jw7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:48.242675 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.242652    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-token" pod "fb26c061-10c7-4d0c-896c-1fbd27ed00aa" (UID: "fb26c061-10c7-4d0c-896c-1fbd27ed00aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:48.242747 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.242681    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-istio-envoy" pod "166343e8-524c-4668-9094-a503991be4f9" (UID: "166343e8-524c-4668-9094-a503991be4f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:48.242747 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.242709    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/166343e8-524c-4668-9094-a503991be4f9-istio-token" pod "166343e8-524c-4668-9094-a503991be4f9" (UID: "166343e8-524c-4668-9094-a503991be4f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:48.243294 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.243272    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-kube-api-access-bkqjq" pod "fb26c061-10c7-4d0c-896c-1fbd27ed00aa" (UID: "fb26c061-10c7-4d0c-896c-1fbd27ed00aa"). InnerVolumeSpecName "kube-api-access-bkqjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:48.243294 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.243279    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-envoy" pod "fb26c061-10c7-4d0c-896c-1fbd27ed00aa" (UID: "fb26c061-10c7-4d0c-896c-1fbd27ed00aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:48.243572 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.243553    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/166343e8-524c-4668-9094-a503991be4f9-istio-podinfo" pod "166343e8-524c-4668-9094-a503991be4f9" (UID: "166343e8-524c-4668-9094-a503991be4f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:00:48.243854 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.243839    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-podinfo" pod "fb26c061-10c7-4d0c-896c-1fbd27ed00aa" (UID: "fb26c061-10c7-4d0c-896c-1fbd27ed00aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:00:48.343283 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.343211    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.343283 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.343263    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/166343e8-524c-4668-9094-a503991be4f9-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.343283 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.343280    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jw7k\" (UniqueName: \"kubernetes.io/projected/166343e8-524c-4668-9094-a503991be4f9-kube-api-access-4jw7k\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.344500 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.343296    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.344500 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.343309    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.344500 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.343324    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.344500 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.343338    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/166343e8-524c-4668-9094-a503991be4f9-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.344500 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.343354    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.344500 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.343367    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/166343e8-524c-4668-9094-a503991be4f9-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.344500 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.343381    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/166343e8-524c-4668-9094-a503991be4f9-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.344500 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.343396    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.344500 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.343409    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkqjq\" (UniqueName: \"kubernetes.io/projected/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-kube-api-access-bkqjq\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.344500 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.343423    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.344500 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.343437    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.344500 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.343450    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb26c061-10c7-4d0c-896c-1fbd27ed00aa-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.348436 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.348091    5875 patch_prober.go:29] interesting pod/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 02:00:48.348436 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.348148    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45" podUID="77d448cc-8da2-4cbc-9d7e-240e51a096c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 02:00:48.595490 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.595413    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 02:00:48.600775 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.600750    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 02:00:48.638829 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.638806    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 02:00:48.663526 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.663497    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-workload-certs\") pod \"be86e257-8a07-418f-a12a-339e09651c93\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") "
Nov 02 02:00:48.663700 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.663591    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-data\") pod \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") "
Nov 02 02:00:48.663700 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.663628    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be86e257-8a07-418f-a12a-339e09651c93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be86e257-8a07-418f-a12a-339e09651c93-istio-podinfo\") pod \"be86e257-8a07-418f-a12a-339e09651c93\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") "
Nov 02 02:00:48.663700 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.663697    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-envoy\") pod \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") "
Nov 02 02:00:48.663870 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.663764    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-workload-certs" pod "be86e257-8a07-418f-a12a-339e09651c93" (UID: "be86e257-8a07-418f-a12a-339e09651c93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:48.663870 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.663804    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-podinfo\") pod \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") "
Nov 02 02:00:48.663870 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.663844    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-credential-socket\") pod \"be86e257-8a07-418f-a12a-339e09651c93\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") "
Nov 02 02:00:48.664023 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.663892    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-credential-socket\") pod \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") "
Nov 02 02:00:48.664023 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.663925    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-data" pod "ff6cb777-acc6-4c58-8302-26d6e2906cdf" (UID: "ff6cb777-acc6-4c58-8302-26d6e2906cdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:48.664023 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.663934    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-token\" (UniqueName: \"kubernetes.io/projected/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-token\") pod \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") "
Nov 02 02:00:48.664023 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.663991    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-workload-certs\") pod \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") "
Nov 02 02:00:48.664308 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.664078    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-istio-envoy\") pod \"be86e257-8a07-418f-a12a-339e09651c93\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") "
Nov 02 02:00:48.664308 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.664093    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-credential-socket" pod "be86e257-8a07-418f-a12a-339e09651c93" (UID: "be86e257-8a07-418f-a12a-339e09651c93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:48.664308 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.664197    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-workload-socket\") pod \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") "
Nov 02 02:00:48.664308 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.664255    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be86e257-8a07-418f-a12a-339e09651c93-kube-api-access-kmb4j\" (UniqueName: \"kubernetes.io/projected/be86e257-8a07-418f-a12a-339e09651c93-kube-api-access-kmb4j\") pod \"be86e257-8a07-418f-a12a-339e09651c93\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") "
Nov 02 02:00:48.664308 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.664270    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-credential-socket" pod "ff6cb777-acc6-4c58-8302-26d6e2906cdf" (UID: "ff6cb777-acc6-4c58-8302-26d6e2906cdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:48.664308 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.664286    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-workload-socket\") pod \"be86e257-8a07-418f-a12a-339e09651c93\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") "
Nov 02 02:00:48.664605 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.664334    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-workload-certs" pod "ff6cb777-acc6-4c58-8302-26d6e2906cdf" (UID: "ff6cb777-acc6-4c58-8302-26d6e2906cdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:48.664605 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.664350    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-istio-data\") pod \"be86e257-8a07-418f-a12a-339e09651c93\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") "
Nov 02 02:00:48.664605 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.664412    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istiod-ca-cert\") pod \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") "
Nov 02 02:00:48.664605 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.664452    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be86e257-8a07-418f-a12a-339e09651c93-istio-token\" (UniqueName: \"kubernetes.io/projected/be86e257-8a07-418f-a12a-339e09651c93-istio-token\") pod \"be86e257-8a07-418f-a12a-339e09651c93\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") "
Nov 02 02:00:48.664605 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.664457    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-workload-socket" pod "ff6cb777-acc6-4c58-8302-26d6e2906cdf" (UID: "ff6cb777-acc6-4c58-8302-26d6e2906cdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:48.664605 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.664507    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be86e257-8a07-418f-a12a-339e09651c93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be86e257-8a07-418f-a12a-339e09651c93-istiod-ca-cert\") pod \"be86e257-8a07-418f-a12a-339e09651c93\" (UID: \"be86e257-8a07-418f-a12a-339e09651c93\") "
Nov 02 02:00:48.664605 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.664539    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff6cb777-acc6-4c58-8302-26d6e2906cdf-kube-api-access-t2k7d\" (UniqueName: \"kubernetes.io/projected/ff6cb777-acc6-4c58-8302-26d6e2906cdf-kube-api-access-t2k7d\") pod \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\" (UID: \"ff6cb777-acc6-4c58-8302-26d6e2906cdf\") "
Nov 02 02:00:48.665166 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.664633    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-istio-data" pod "be86e257-8a07-418f-a12a-339e09651c93" (UID: "be86e257-8a07-418f-a12a-339e09651c93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:48.665166 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.664806    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-workload-socket" pod "be86e257-8a07-418f-a12a-339e09651c93" (UID: "be86e257-8a07-418f-a12a-339e09651c93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:48.665473 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.665438    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be86e257-8a07-418f-a12a-339e09651c93-istiod-ca-cert" pod "be86e257-8a07-418f-a12a-339e09651c93" (UID: "be86e257-8a07-418f-a12a-339e09651c93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:48.666075 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.666049    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istiod-ca-cert" pod "ff6cb777-acc6-4c58-8302-26d6e2906cdf" (UID: "ff6cb777-acc6-4c58-8302-26d6e2906cdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:48.666873 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.666851    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-envoy" pod "ff6cb777-acc6-4c58-8302-26d6e2906cdf" (UID: "ff6cb777-acc6-4c58-8302-26d6e2906cdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:48.666987 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.666922    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-podinfo" pod "ff6cb777-acc6-4c58-8302-26d6e2906cdf" (UID: "ff6cb777-acc6-4c58-8302-26d6e2906cdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:00:48.667055 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.667003    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-token" pod "ff6cb777-acc6-4c58-8302-26d6e2906cdf" (UID: "ff6cb777-acc6-4c58-8302-26d6e2906cdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:48.667288 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.667262    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be86e257-8a07-418f-a12a-339e09651c93-istio-podinfo" pod "be86e257-8a07-418f-a12a-339e09651c93" (UID: "be86e257-8a07-418f-a12a-339e09651c93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:00:48.667548 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.667454    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.667548 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.667470    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be86e257-8a07-418f-a12a-339e09651c93-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.667548 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.667483    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.667548 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.667491    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.667548 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.667500    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.667548 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.667508    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.667548 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.667518    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.667548 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.667526    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.667548 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.667534    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6cb777-acc6-4c58-8302-26d6e2906cdf-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.667548 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.667541    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.667548 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.667550    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.668150 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.667560    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff6cb777-acc6-4c58-8302-26d6e2906cdf-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.668150 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.667568    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be86e257-8a07-418f-a12a-339e09651c93-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.668150 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.667576    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.668150 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.668114    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6cb777-acc6-4c58-8302-26d6e2906cdf-kube-api-access-t2k7d" pod "ff6cb777-acc6-4c58-8302-26d6e2906cdf" (UID: "ff6cb777-acc6-4c58-8302-26d6e2906cdf"). InnerVolumeSpecName "kube-api-access-t2k7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:48.668416 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.668359    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-istio-envoy" pod "be86e257-8a07-418f-a12a-339e09651c93" (UID: "be86e257-8a07-418f-a12a-339e09651c93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:48.668945 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.668929    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be86e257-8a07-418f-a12a-339e09651c93-istio-token" pod "be86e257-8a07-418f-a12a-339e09651c93" (UID: "be86e257-8a07-418f-a12a-339e09651c93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:48.670653 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.670632    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be86e257-8a07-418f-a12a-339e09651c93-kube-api-access-kmb4j" pod "be86e257-8a07-418f-a12a-339e09651c93" (UID: "be86e257-8a07-418f-a12a-339e09651c93"). InnerVolumeSpecName "kube-api-access-kmb4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:48.768376 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.768349    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-data\") pod \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") "
Nov 02 02:00:48.768537 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.768428    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-credential-socket\") pod \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") "
Nov 02 02:00:48.768537 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.768453    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-token\" (UniqueName: \"kubernetes.io/projected/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-token\") pod \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") "
Nov 02 02:00:48.768664 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.768551    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istiod-ca-cert\") pod \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") "
Nov 02 02:00:48.768664 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.768657    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-workload-socket\") pod \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") "
Nov 02 02:00:48.768765 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.768696    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-data" pod "77d448cc-8da2-4cbc-9d7e-240e51a096c0" (UID: "77d448cc-8da2-4cbc-9d7e-240e51a096c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:48.768765 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.768716    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-credential-socket" pod "77d448cc-8da2-4cbc-9d7e-240e51a096c0" (UID: "77d448cc-8da2-4cbc-9d7e-240e51a096c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:48.768765 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.768751    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-workload-certs\") pod \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") "
Nov 02 02:00:48.768878 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.768827    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77d448cc-8da2-4cbc-9d7e-240e51a096c0-kube-api-access-7mq7m\" (UniqueName: \"kubernetes.io/projected/77d448cc-8da2-4cbc-9d7e-240e51a096c0-kube-api-access-7mq7m\") pod \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") "
Nov 02 02:00:48.768878 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.768869    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-envoy\") pod \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") "
Nov 02 02:00:48.768971 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.768915    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-podinfo\") pod \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\" (UID: \"77d448cc-8da2-4cbc-9d7e-240e51a096c0\") "
Nov 02 02:00:48.769031 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.769003    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-workload-socket" pod "77d448cc-8da2-4cbc-9d7e-240e51a096c0" (UID: "77d448cc-8da2-4cbc-9d7e-240e51a096c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:48.769108 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.769085    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-workload-certs" pod "77d448cc-8da2-4cbc-9d7e-240e51a096c0" (UID: "77d448cc-8da2-4cbc-9d7e-240e51a096c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:48.769776 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.769681    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be86e257-8a07-418f-a12a-339e09651c93-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.769776 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.769706    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.769776 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.769722    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmb4j\" (UniqueName: \"kubernetes.io/projected/be86e257-8a07-418f-a12a-339e09651c93-kube-api-access-kmb4j\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.769776 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.769736    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.769776 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.769750    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be86e257-8a07-418f-a12a-339e09651c93-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.769776 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.769764    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2k7d\" (UniqueName: \"kubernetes.io/projected/ff6cb777-acc6-4c58-8302-26d6e2906cdf-kube-api-access-t2k7d\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.769776 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.769777    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.770259 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.769790    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.770468 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.770349    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istiod-ca-cert" pod "77d448cc-8da2-4cbc-9d7e-240e51a096c0" (UID: "77d448cc-8da2-4cbc-9d7e-240e51a096c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:48.772302 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.772276    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d448cc-8da2-4cbc-9d7e-240e51a096c0-kube-api-access-7mq7m" pod "77d448cc-8da2-4cbc-9d7e-240e51a096c0" (UID: "77d448cc-8da2-4cbc-9d7e-240e51a096c0"). InnerVolumeSpecName "kube-api-access-7mq7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:48.772933 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.772912    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-envoy" pod "77d448cc-8da2-4cbc-9d7e-240e51a096c0" (UID: "77d448cc-8da2-4cbc-9d7e-240e51a096c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:48.773374 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.773348    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-token" pod "77d448cc-8da2-4cbc-9d7e-240e51a096c0" (UID: "77d448cc-8da2-4cbc-9d7e-240e51a096c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:48.773532 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.773510    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-podinfo" pod "77d448cc-8da2-4cbc-9d7e-240e51a096c0" (UID: "77d448cc-8da2-4cbc-9d7e-240e51a096c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:00:48.871444 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.871355    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.871444 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.871395    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mq7m\" (UniqueName: \"kubernetes.io/projected/77d448cc-8da2-4cbc-9d7e-240e51a096c0-kube-api-access-7mq7m\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.871444 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.871407    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.871444 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.871438    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.871782 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.871451    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d448cc-8da2-4cbc-9d7e-240e51a096c0-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:48.986872 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.986829    5875 generic.go:356] "Generic (PLEG): container finished" podID="fb26c061-10c7-4d0c-896c-1fbd27ed00aa" containerID="3e906e3ea143f21d04aa923cab25c201ce66e79724cc807bab6c10f3a876d9af" exitCode=0
Nov 02 02:00:48.987410 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.986969    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"
Nov 02 02:00:48.987410 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.986969    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph" event={"ID":"fb26c061-10c7-4d0c-896c-1fbd27ed00aa","Type":"ContainerDied","Data":"3e906e3ea143f21d04aa923cab25c201ce66e79724cc807bab6c10f3a876d9af"}
Nov 02 02:00:48.987410 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.987096    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph" event={"ID":"fb26c061-10c7-4d0c-896c-1fbd27ed00aa","Type":"ContainerDied","Data":"c0fa4753e493aebc6e5e966d0901e79a92954614455c7a2f1a03c07b31332d44"}
Nov 02 02:00:48.987410 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.987125    5875 scope.go:122] "RemoveContainer" containerID="3e906e3ea143f21d04aa923cab25c201ce66e79724cc807bab6c10f3a876d9af"
Nov 02 02:00:48.989963 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.989930    5875 generic.go:356] "Generic (PLEG): container finished" podID="77d448cc-8da2-4cbc-9d7e-240e51a096c0" containerID="896a02bf2e1970079559dba3d178c18f3bdab2c34502fc0c79d37e0fe13f0be5" exitCode=0
Nov 02 02:00:48.990101 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.990018    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45" event={"ID":"77d448cc-8da2-4cbc-9d7e-240e51a096c0","Type":"ContainerDied","Data":"896a02bf2e1970079559dba3d178c18f3bdab2c34502fc0c79d37e0fe13f0be5"}
Nov 02 02:00:48.990101 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.990062    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45" event={"ID":"77d448cc-8da2-4cbc-9d7e-240e51a096c0","Type":"ContainerDied","Data":"006eca3cfe18d7327f84160914f0a95b5404a61d89e213593da28cefe98530a5"}
Nov 02 02:00:48.990214 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.990144    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"
Nov 02 02:00:48.996483 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.996461    5875 generic.go:356] "Generic (PLEG): container finished" podID="9a054e97-d2a9-47fd-a380-24f4c97d908e" containerID="19244c9bd784f673fa14b8bef85d37ceeefef9c38799461dfde948ae773e2ee0" exitCode=0
Nov 02 02:00:48.996609 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.996542    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg" event={"ID":"9a054e97-d2a9-47fd-a380-24f4c97d908e","Type":"ContainerDied","Data":"19244c9bd784f673fa14b8bef85d37ceeefef9c38799461dfde948ae773e2ee0"}
Nov 02 02:00:48.997013 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.996989    5875 scope.go:122] "RemoveContainer" containerID="19244c9bd784f673fa14b8bef85d37ceeefef9c38799461dfde948ae773e2ee0"
Nov 02 02:00:48.998314 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.998297    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"
Nov 02 02:00:48.998546 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.998359    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb" event={"ID":"ff6cb777-acc6-4c58-8302-26d6e2906cdf","Type":"ContainerDied","Data":"3552f1e88dd16e273e586c8388d1fdea3c38a80c89e9c35c857288af4df5001b"}
Nov 02 02:00:48.999679 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.999641    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:00:48.999879 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:48.999856    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:00:49.000037 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.000010    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:00:49.000348 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.000205    5875 generic.go:356] "Generic (PLEG): container finished" podID="be86e257-8a07-418f-a12a-339e09651c93" containerID="3a8c2fd55e1d3b8c9b92e4c787133742599a5d43de66e20ab3e5c1c73d532d1c" exitCode=0
Nov 02 02:00:49.000348 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.000312    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"
Nov 02 02:00:49.000348 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.000327    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj" event={"ID":"be86e257-8a07-418f-a12a-339e09651c93","Type":"ContainerDied","Data":"3a8c2fd55e1d3b8c9b92e4c787133742599a5d43de66e20ab3e5c1c73d532d1c"}
Nov 02 02:00:49.000520 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.000369    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj" event={"ID":"be86e257-8a07-418f-a12a-339e09651c93","Type":"ContainerDied","Data":"5cd645a44a6aa7bf64b15b360ca0ee9046d8d9c3cd48b130eb16b0fa7ba058a2"}
Nov 02 02:00:49.003038 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.003019    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"
Nov 02 02:00:49.003479 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.003053    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw" event={"ID":"166343e8-524c-4668-9094-a503991be4f9","Type":"ContainerDied","Data":"b2465df631c50378199cfb56d7a4ad8de3bb3dd3b35f37dd702fa6df8dfab655"}
Nov 02 02:00:49.008983 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.008967    5875 scope.go:122] "RemoveContainer" containerID="3e906e3ea143f21d04aa923cab25c201ce66e79724cc807bab6c10f3a876d9af"
Nov 02 02:00:49.009336 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:00:49.009312    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e906e3ea143f21d04aa923cab25c201ce66e79724cc807bab6c10f3a876d9af\": container with ID starting with 3e906e3ea143f21d04aa923cab25c201ce66e79724cc807bab6c10f3a876d9af not found: ID does not exist" containerID="3e906e3ea143f21d04aa923cab25c201ce66e79724cc807bab6c10f3a876d9af"
Nov 02 02:00:49.009427 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.009347    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e906e3ea143f21d04aa923cab25c201ce66e79724cc807bab6c10f3a876d9af"} err="failed to get container status \"3e906e3ea143f21d04aa923cab25c201ce66e79724cc807bab6c10f3a876d9af\": rpc error: code = NotFound desc = could not find container \"3e906e3ea143f21d04aa923cab25c201ce66e79724cc807bab6c10f3a876d9af\": container with ID starting with 3e906e3ea143f21d04aa923cab25c201ce66e79724cc807bab6c10f3a876d9af not found: ID does not exist"
Nov 02 02:00:49.009427 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.009368    5875 scope.go:122] "RemoveContainer" containerID="896a02bf2e1970079559dba3d178c18f3bdab2c34502fc0c79d37e0fe13f0be5"
Nov 02 02:00:49.025592 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.025205    5875 scope.go:122] "RemoveContainer" containerID="896a02bf2e1970079559dba3d178c18f3bdab2c34502fc0c79d37e0fe13f0be5"
Nov 02 02:00:49.026048 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:00:49.026012    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"896a02bf2e1970079559dba3d178c18f3bdab2c34502fc0c79d37e0fe13f0be5\": container with ID starting with 896a02bf2e1970079559dba3d178c18f3bdab2c34502fc0c79d37e0fe13f0be5 not found: ID does not exist" containerID="896a02bf2e1970079559dba3d178c18f3bdab2c34502fc0c79d37e0fe13f0be5"
Nov 02 02:00:49.026148 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.026059    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"896a02bf2e1970079559dba3d178c18f3bdab2c34502fc0c79d37e0fe13f0be5"} err="failed to get container status \"896a02bf2e1970079559dba3d178c18f3bdab2c34502fc0c79d37e0fe13f0be5\": rpc error: code = NotFound desc = could not find container \"896a02bf2e1970079559dba3d178c18f3bdab2c34502fc0c79d37e0fe13f0be5\": container with ID starting with 896a02bf2e1970079559dba3d178c18f3bdab2c34502fc0c79d37e0fe13f0be5 not found: ID does not exist"
Nov 02 02:00:49.026148 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.026082    5875 scope.go:122] "RemoveContainer" containerID="97de7d64ce55692ccda83e9d5866414b1d18cf9d5fc60159179055c4e5e2207a"
Nov 02 02:00:49.052848 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.052829    5875 scope.go:122] "RemoveContainer" containerID="3a8c2fd55e1d3b8c9b92e4c787133742599a5d43de66e20ab3e5c1c73d532d1c"
Nov 02 02:00:49.067166 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.067145    5875 scope.go:122] "RemoveContainer" containerID="3a8c2fd55e1d3b8c9b92e4c787133742599a5d43de66e20ab3e5c1c73d532d1c"
Nov 02 02:00:49.067505 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:00:49.067484    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a8c2fd55e1d3b8c9b92e4c787133742599a5d43de66e20ab3e5c1c73d532d1c\": container with ID starting with 3a8c2fd55e1d3b8c9b92e4c787133742599a5d43de66e20ab3e5c1c73d532d1c not found: ID does not exist" containerID="3a8c2fd55e1d3b8c9b92e4c787133742599a5d43de66e20ab3e5c1c73d532d1c"
Nov 02 02:00:49.067590 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.067510    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a8c2fd55e1d3b8c9b92e4c787133742599a5d43de66e20ab3e5c1c73d532d1c"} err="failed to get container status \"3a8c2fd55e1d3b8c9b92e4c787133742599a5d43de66e20ab3e5c1c73d532d1c\": rpc error: code = NotFound desc = could not find container \"3a8c2fd55e1d3b8c9b92e4c787133742599a5d43de66e20ab3e5c1c73d532d1c\": container with ID starting with 3a8c2fd55e1d3b8c9b92e4c787133742599a5d43de66e20ab3e5c1c73d532d1c not found: ID does not exist"
Nov 02 02:00:49.067590 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.067529    5875 scope.go:122] "RemoveContainer" containerID="e363896987f9192d328bf8d611238d41fa218712e4fa3794af11dbb506c6843a"
Nov 02 02:00:49.161075 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.160986    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"]
Nov 02 02:00:49.166479 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.166454    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-n8bph"]
Nov 02 02:00:49.272809 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.272778    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"]
Nov 02 02:00:49.284858 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.284831    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-kpm45"]
Nov 02 02:00:49.397886 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.397847    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"]
Nov 02 02:00:49.401332 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.401307    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-af807eae-openshift-default-66fcb6ff5c-nb7bb"]
Nov 02 02:00:49.510125 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.509898    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"]
Nov 02 02:00:49.514450 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.514425    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-rg2vw"]
Nov 02 02:00:49.761119 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.761018    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"]
Nov 02 02:00:49.768473 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:49.768446    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-061fab-openshift-default-d6b5b6c67-2t6pj"]
Nov 02 02:00:50.038621 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:50.038542    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg" event={"ID":"9a054e97-d2a9-47fd-a380-24f4c97d908e","Type":"ContainerStarted","Data":"aa11d9cba343b58e066c24492392dde2d37937699ea3b8a4c99027184bdf4147"}
Nov 02 02:00:50.701268 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:50.698824    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="166343e8-524c-4668-9094-a503991be4f9" path="/var/lib/kubelet/pods/166343e8-524c-4668-9094-a503991be4f9/volumes"
Nov 02 02:00:50.728247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:50.726684    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77d448cc-8da2-4cbc-9d7e-240e51a096c0" path="/var/lib/kubelet/pods/77d448cc-8da2-4cbc-9d7e-240e51a096c0/volumes"
Nov 02 02:00:50.728247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:50.727421    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be86e257-8a07-418f-a12a-339e09651c93" path="/var/lib/kubelet/pods/be86e257-8a07-418f-a12a-339e09651c93/volumes"
Nov 02 02:00:50.728247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:50.728072    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb26c061-10c7-4d0c-896c-1fbd27ed00aa" path="/var/lib/kubelet/pods/fb26c061-10c7-4d0c-896c-1fbd27ed00aa/volumes"
Nov 02 02:00:50.731774 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:50.731748    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff6cb777-acc6-4c58-8302-26d6e2906cdf" path="/var/lib/kubelet/pods/ff6cb777-acc6-4c58-8302-26d6e2906cdf/volumes"
Nov 02 02:00:50.735294 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:50.735267    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 02:00:50.735530 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:50.735509    5875 patch_prober.go:29] interesting pod/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 02 02:00:50.735633 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:50.735561    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg" podUID="9a054e97-d2a9-47fd-a380-24f4c97d908e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 02 02:00:51.735621 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:51.735583    5875 patch_prober.go:29] interesting pod/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 02 02:00:51.735995 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:51.735634    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg" podUID="9a054e97-d2a9-47fd-a380-24f4c97d908e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 02 02:00:52.736238 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:52.736191    5875 patch_prober.go:29] interesting pod/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 02 02:00:52.736649 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:52.736288    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg" podUID="9a054e97-d2a9-47fd-a380-24f4c97d908e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 02 02:00:53.736272 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:53.736203    5875 patch_prober.go:29] interesting pod/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 02 02:00:53.736667 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:53.736293    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg" podUID="9a054e97-d2a9-47fd-a380-24f4c97d908e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 02 02:00:53.912509 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:53.912458    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7"]
Nov 02 02:00:53.923109 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:53.913912    5875 cpu_manager.go:414] "containerMap: removing container" podUID="77d448cc-8da2-4cbc-9d7e-240e51a096c0" containerName="istio-proxy"
Nov 02 02:00:53.923464 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:53.923437    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77d448cc-8da2-4cbc-9d7e-240e51a096c0" containerName="istio-proxy"
Nov 02 02:00:53.923585 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:53.923572    5875 cpu_manager.go:414] "containerMap: removing container" podUID="166343e8-524c-4668-9094-a503991be4f9" containerName="istio-proxy"
Nov 02 02:00:53.923674 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:53.923661    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="166343e8-524c-4668-9094-a503991be4f9" containerName="istio-proxy"
Nov 02 02:00:53.923771 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:53.923759    5875 cpu_manager.go:414] "containerMap: removing container" podUID="ff6cb777-acc6-4c58-8302-26d6e2906cdf" containerName="istio-proxy"
Nov 02 02:00:53.923857 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:53.923845    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff6cb777-acc6-4c58-8302-26d6e2906cdf" containerName="istio-proxy"
Nov 02 02:00:53.924016 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:53.924004    5875 cpu_manager.go:414] "containerMap: removing container" podUID="ff6cb777-acc6-4c58-8302-26d6e2906cdf" containerName="istio-proxy"
Nov 02 02:00:53.924105 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:53.924093    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff6cb777-acc6-4c58-8302-26d6e2906cdf" containerName="istio-proxy"
Nov 02 02:00:53.924254 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:53.924240    5875 cpu_manager.go:414] "containerMap: removing container" podUID="fb26c061-10c7-4d0c-896c-1fbd27ed00aa" containerName="istio-proxy"
Nov 02 02:00:53.924356 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:53.924344    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb26c061-10c7-4d0c-896c-1fbd27ed00aa" containerName="istio-proxy"
Nov 02 02:00:53.924437 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:53.924426    5875 cpu_manager.go:414] "containerMap: removing container" podUID="be86e257-8a07-418f-a12a-339e09651c93" containerName="istio-proxy"
Nov 02 02:00:53.924515 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:53.924504    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be86e257-8a07-418f-a12a-339e09651c93" containerName="istio-proxy"
Nov 02 02:00:53.924995 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:53.924928    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="77d448cc-8da2-4cbc-9d7e-240e51a096c0" containerName="istio-proxy"
Nov 02 02:00:53.931297 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:53.931271    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="166343e8-524c-4668-9094-a503991be4f9" containerName="istio-proxy"
Nov 02 02:00:53.931485 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:53.931470    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff6cb777-acc6-4c58-8302-26d6e2906cdf" containerName="istio-proxy"
Nov 02 02:00:53.931604 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:53.931591    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff6cb777-acc6-4c58-8302-26d6e2906cdf" containerName="istio-proxy"
Nov 02 02:00:53.931694 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:53.931683    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="166343e8-524c-4668-9094-a503991be4f9" containerName="istio-proxy"
Nov 02 02:00:53.931783 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:53.931773    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb26c061-10c7-4d0c-896c-1fbd27ed00aa" containerName="istio-proxy"
Nov 02 02:00:53.931871 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:53.931861    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="be86e257-8a07-418f-a12a-339e09651c93" containerName="istio-proxy"
Nov 02 02:00:54.001771 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:53.989321    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7"
Nov 02 02:00:54.070362 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:54.070327    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-8e79d7-dockercfg-zvssc\""
Nov 02 02:00:54.169245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:54.164676    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade83e3d-3ae8-4732-9a85-955c54135ead-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7\" (UID: \"ade83e3d-3ae8-4732-9a85-955c54135ead\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7"
Nov 02 02:00:54.169245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:54.164894    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjbfv\" (UniqueName: \"kubernetes.io/projected/ade83e3d-3ae8-4732-9a85-955c54135ead-kube-api-access-qjbfv\") pod \"payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7\" (UID: \"ade83e3d-3ae8-4732-9a85-955c54135ead\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7"
Nov 02 02:00:54.184283 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:54.171529    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-8e79d7\""
Nov 02 02:00:54.276989 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:54.276873    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade83e3d-3ae8-4732-9a85-955c54135ead-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7\" (UID: \"ade83e3d-3ae8-4732-9a85-955c54135ead\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7"
Nov 02 02:00:54.285127 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:54.285090    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjbfv\" (UniqueName: \"kubernetes.io/projected/ade83e3d-3ae8-4732-9a85-955c54135ead-kube-api-access-qjbfv\") pod \"payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7\" (UID: \"ade83e3d-3ae8-4732-9a85-955c54135ead\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7"
Nov 02 02:00:54.291081 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:54.291031    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7"]
Nov 02 02:00:54.293397 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:54.293372    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj"]
Nov 02 02:00:54.312045 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:54.312007    5875 cpu_manager.go:414] "containerMap: removing container" podUID="166343e8-524c-4668-9094-a503991be4f9" containerName="istio-proxy"
Nov 02 02:00:54.312303 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:54.312284    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="166343e8-524c-4668-9094-a503991be4f9" containerName="istio-proxy"
Nov 02 02:00:54.349237 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:54.349183    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade83e3d-3ae8-4732-9a85-955c54135ead-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7\" (UID: \"ade83e3d-3ae8-4732-9a85-955c54135ead\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7"
Nov 02 02:00:54.353638 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:54.353580    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj"
Nov 02 02:00:54.391244 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:54.389385    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj"]
Nov 02 02:00:54.395245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:54.391700    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32eed2d4-07e6-41ba-885c-438ddfe308c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj\" (UID: \"32eed2d4-07e6-41ba-885c-438ddfe308c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj"
Nov 02 02:00:54.395245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:54.391976    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msz7b\" (UniqueName: \"kubernetes.io/projected/32eed2d4-07e6-41ba-885c-438ddfe308c7-kube-api-access-msz7b\") pod \"payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj\" (UID: \"32eed2d4-07e6-41ba-885c-438ddfe308c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj"
Nov 02 02:00:54.458491 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:54.458451    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjbfv\" (UniqueName: \"kubernetes.io/projected/ade83e3d-3ae8-4732-9a85-955c54135ead-kube-api-access-qjbfv\") pod \"payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7\" (UID: \"ade83e3d-3ae8-4732-9a85-955c54135ead\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7"
Nov 02 02:00:54.531418 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:54.531313    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32eed2d4-07e6-41ba-885c-438ddfe308c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj\" (UID: \"32eed2d4-07e6-41ba-885c-438ddfe308c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj"
Nov 02 02:00:54.531607 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:54.531583    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msz7b\" (UniqueName: \"kubernetes.io/projected/32eed2d4-07e6-41ba-885c-438ddfe308c7-kube-api-access-msz7b\") pod \"payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj\" (UID: \"32eed2d4-07e6-41ba-885c-438ddfe308c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj"
Nov 02 02:00:54.532294 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:54.532268    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32eed2d4-07e6-41ba-885c-438ddfe308c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj\" (UID: \"32eed2d4-07e6-41ba-885c-438ddfe308c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj"
Nov 02 02:00:54.592758 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:54.592731    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msz7b\" (UniqueName: \"kubernetes.io/projected/32eed2d4-07e6-41ba-885c-438ddfe308c7-kube-api-access-msz7b\") pod \"payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj\" (UID: \"32eed2d4-07e6-41ba-885c-438ddfe308c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj"
Nov 02 02:00:54.668394 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:54.668350    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7"
Nov 02 02:00:54.736725 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:54.736690    5875 patch_prober.go:29] interesting pod/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 02 02:00:54.737190 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:54.736752    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg" podUID="9a054e97-d2a9-47fd-a380-24f4c97d908e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 02 02:00:54.772668 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:54.772640    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj"
Nov 02 02:00:55.075300 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:55.075271    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7"]
Nov 02 02:00:55.076379 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:00:55.076353    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade83e3d_3ae8_4732_9a85_955c54135ead.slice/crio-88b11d86c5a2c605fed8fa3bcc7874f071a646219469127df7d11ff861ed7273 WatchSource:0}: Error finding container 88b11d86c5a2c605fed8fa3bcc7874f071a646219469127df7d11ff861ed7273: Status 404 returned error can't find the container with id 88b11d86c5a2c605fed8fa3bcc7874f071a646219469127df7d11ff861ed7273
Nov 02 02:00:55.186439 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:55.186413    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj"]
Nov 02 02:00:55.187151 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:00:55.187125    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32eed2d4_07e6_41ba_885c_438ddfe308c7.slice/crio-df22a917258e9f29236a9128ab08342eb94fe18f3eca8696a225233ddeeed259 WatchSource:0}: Error finding container df22a917258e9f29236a9128ab08342eb94fe18f3eca8696a225233ddeeed259: Status 404 returned error can't find the container with id df22a917258e9f29236a9128ab08342eb94fe18f3eca8696a225233ddeeed259
Nov 02 02:00:55.357813 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:55.357718    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj" event={"ID":"32eed2d4-07e6-41ba-885c-438ddfe308c7","Type":"ContainerStarted","Data":"1d06727cb9ffb0526fafdbefcb0b1104bfd4904e4fe69806f2aee7002c1a960a"}
Nov 02 02:00:55.357813 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:55.357782    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj" event={"ID":"32eed2d4-07e6-41ba-885c-438ddfe308c7","Type":"ContainerStarted","Data":"df22a917258e9f29236a9128ab08342eb94fe18f3eca8696a225233ddeeed259"}
Nov 02 02:00:55.358050 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:55.357866    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj"
Nov 02 02:00:55.360044 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:55.360003    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7" event={"ID":"ade83e3d-3ae8-4732-9a85-955c54135ead","Type":"ContainerStarted","Data":"59557de22a314f21b060ba06168ee32eb40d862fcbbea6c08c0ed211cdd81821"}
Nov 02 02:00:55.360164 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:55.360060    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7" event={"ID":"ade83e3d-3ae8-4732-9a85-955c54135ead","Type":"ContainerStarted","Data":"88b11d86c5a2c605fed8fa3bcc7874f071a646219469127df7d11ff861ed7273"}
Nov 02 02:00:55.360164 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:55.360098    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7"
Nov 02 02:00:55.476236 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:55.476179    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj" podStartSLOduration=1.4761636 podStartE2EDuration="1.4761636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:55.472448523 +0000 UTC m=+1702.913485058" watchObservedRunningTime="2026-11-02 02:00:55.4761636 +0000 UTC m=+1702.917200096"
Nov 02 02:00:55.607311 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:55.603939    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7" podStartSLOduration=2.6039161269999997 podStartE2EDuration="2.603916127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:55.601869951 +0000 UTC m=+1703.042906449" watchObservedRunningTime="2026-11-02 02:00:55.603916127 +0000 UTC m=+1703.044952626"
Nov 02 02:00:55.787326 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:55.760447    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 02:00:55.787326 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:55.772616    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 02:00:55.787326 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:55.786757    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 02:00:56.551263 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:56.550349    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz"]
Nov 02 02:00:56.567246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:56.558049    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz"
Nov 02 02:00:56.609251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:56.605210    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"]
Nov 02 02:00:56.609251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:56.605558    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f" podUID="3f4aa706-493e-42ec-a124-187020d8aa3e" containerName="istio-proxy" containerID="cri-o://ec384196885d2de31d686f8701f35bad124fcebbd5b54b9b32f356131a18677e" gracePeriod=30
Nov 02 02:00:56.629654 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:56.617116    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"]
Nov 02 02:00:56.629654 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:56.617458    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n" podUID="6b0c4e37-e032-4b48-8261-81294adf8635" containerName="istio-proxy" containerID="cri-o://9b8d5f60b15c963b43d64fec47ac742c52678a4f5f9df8c5dcb09f64c73ce6a7" gracePeriod=30
Nov 02 02:00:56.674246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:56.673312    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-c1aad5-dockercfg-6ct9t\""
Nov 02 02:00:56.674246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:56.673407    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-c1aad5\""
Nov 02 02:00:56.674246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:56.673654    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"]
Nov 02 02:00:56.674246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:56.673881    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh" podUID="05f7e0b6-dc17-4891-9817-242be8465c54" containerName="istio-proxy" containerID="cri-o://7dcf8805adcd0c037b18a35a45dfc5dc6a190a503be98f61da4dfe2a49e2891b" gracePeriod=30
Nov 02 02:00:56.683244 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:56.680980    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d212576-5a9c-4523-8a0f-93b851be6406-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz\" (UID: \"0d212576-5a9c-4523-8a0f-93b851be6406\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz"
Nov 02 02:00:56.683244 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:56.681065    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz2j8\" (UniqueName: \"kubernetes.io/projected/0d212576-5a9c-4523-8a0f-93b851be6406-kube-api-access-jz2j8\") pod \"payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz\" (UID: \"0d212576-5a9c-4523-8a0f-93b851be6406\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz"
Nov 02 02:00:56.811292 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:56.805411    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d212576-5a9c-4523-8a0f-93b851be6406-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz\" (UID: \"0d212576-5a9c-4523-8a0f-93b851be6406\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz"
Nov 02 02:00:56.811292 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:56.805510    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz2j8\" (UniqueName: \"kubernetes.io/projected/0d212576-5a9c-4523-8a0f-93b851be6406-kube-api-access-jz2j8\") pod \"payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz\" (UID: \"0d212576-5a9c-4523-8a0f-93b851be6406\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz"
Nov 02 02:00:56.811292 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:56.806926    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d212576-5a9c-4523-8a0f-93b851be6406-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz\" (UID: \"0d212576-5a9c-4523-8a0f-93b851be6406\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz"
Nov 02 02:00:56.861278 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:56.857549    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"]
Nov 02 02:00:56.861278 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:56.857813    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7" podUID="3726ebef-c581-4736-b0a3-ae417a6e8414" containerName="istio-proxy" containerID="cri-o://90a1419c7d2b6bbe194675665acef20037e9d57e1e1f3be8fee60b2d03ec4d49" gracePeriod=30
Nov 02 02:00:57.010814 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:57.008448    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz"]
Nov 02 02:00:57.033145 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:57.031144    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz2j8\" (UniqueName: \"kubernetes.io/projected/0d212576-5a9c-4523-8a0f-93b851be6406-kube-api-access-jz2j8\") pod \"payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz\" (UID: \"0d212576-5a9c-4523-8a0f-93b851be6406\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz"
Nov 02 02:00:57.043719 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:57.043608    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb"]
Nov 02 02:00:57.051400 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:57.051378    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb"
Nov 02 02:00:57.051895 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:57.051879    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb"]
Nov 02 02:00:57.145395 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:57.145331    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0656b83a-c10d-4929-b029-c3852202b703-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb\" (UID: \"0656b83a-c10d-4929-b029-c3852202b703\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb"
Nov 02 02:00:57.145608 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:57.145592    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88gnl\" (UniqueName: \"kubernetes.io/projected/0656b83a-c10d-4929-b029-c3852202b703-kube-api-access-88gnl\") pod \"payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb\" (UID: \"0656b83a-c10d-4929-b029-c3852202b703\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb"
Nov 02 02:00:57.209377 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:57.209351    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz"
Nov 02 02:00:57.246772 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:57.246738    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88gnl\" (UniqueName: \"kubernetes.io/projected/0656b83a-c10d-4929-b029-c3852202b703-kube-api-access-88gnl\") pod \"payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb\" (UID: \"0656b83a-c10d-4929-b029-c3852202b703\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb"
Nov 02 02:00:57.246964 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:57.246858    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0656b83a-c10d-4929-b029-c3852202b703-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb\" (UID: \"0656b83a-c10d-4929-b029-c3852202b703\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb"
Nov 02 02:00:57.247748 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:57.247722    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0656b83a-c10d-4929-b029-c3852202b703-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb\" (UID: \"0656b83a-c10d-4929-b029-c3852202b703\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb"
Nov 02 02:00:57.308891 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:57.308857    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88gnl\" (UniqueName: \"kubernetes.io/projected/0656b83a-c10d-4929-b029-c3852202b703-kube-api-access-88gnl\") pod \"payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb\" (UID: \"0656b83a-c10d-4929-b029-c3852202b703\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb"
Nov 02 02:00:57.365934 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:57.365907    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb"
Nov 02 02:00:57.591516 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:57.591486    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz"]
Nov 02 02:00:57.592404 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:00:57.592379    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d212576_5a9c_4523_8a0f_93b851be6406.slice/crio-f29c1c61c860109e943dc8244c253f8770f3bc76312348d24d6acaf71bd07eb8 WatchSource:0}: Error finding container f29c1c61c860109e943dc8244c253f8770f3bc76312348d24d6acaf71bd07eb8: Status 404 returned error can't find the container with id f29c1c61c860109e943dc8244c253f8770f3bc76312348d24d6acaf71bd07eb8
Nov 02 02:00:57.747354 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:57.747327    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb"]
Nov 02 02:00:57.747769 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:00:57.747750    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0656b83a_c10d_4929_b029_c3852202b703.slice/crio-4bb6e3293299a2c76e226551470240fd588f53b56d14bacded6303a8a8ebf7eb WatchSource:0}: Error finding container 4bb6e3293299a2c76e226551470240fd588f53b56d14bacded6303a8a8ebf7eb: Status 404 returned error can't find the container with id 4bb6e3293299a2c76e226551470240fd588f53b56d14bacded6303a8a8ebf7eb
Nov 02 02:00:58.260249 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.254871    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"]
Nov 02 02:00:58.260249 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.255344    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt" podUID="c637ee2b-9c6b-489e-8f6b-49112217a03c" containerName="istio-proxy" containerID="cri-o://ca1bf69d08ce3a5c2eb3ec7b5bfd9869e3244d66b88d50cd5da80b992579494a" gracePeriod=30
Nov 02 02:00:58.273463 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.273420    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"]
Nov 02 02:00:58.273972 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.273923    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw" podUID="bceaf892-6e4e-4472-bf8d-34b1fe161036" containerName="istio-proxy" containerID="cri-o://867fc8e3aacab21720f28a83a60d87a1999b0b1cb75cf4a6e7c6ae051b997ce7" gracePeriod=30
Nov 02 02:00:58.301837 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.301377    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"]
Nov 02 02:00:58.301837 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.301618    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b" podUID="7bef4b3e-a32d-4171-bd35-d19e9feef1e4" containerName="istio-proxy" containerID="cri-o://1408e3ce8a3aea7a7c66569514dc05028c2af17e9ffcce6b620da594484ce89a" gracePeriod=30
Nov 02 02:00:58.313493 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.313463    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"]
Nov 02 02:00:58.313952 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.313920    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k" podUID="5ec9fbc9-a60c-4a82-932c-0af50b66ecab" containerName="istio-proxy" containerID="cri-o://5bbac46e87a02af71b7c6653d29432628bcdf0f3f0ae71c70b984912a2bd30c5" gracePeriod=30
Nov 02 02:00:58.334251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.334119    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:00:58.334531 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.334503    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh" podUID="05f7e0b6-dc17-4891-9817-242be8465c54" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:00:58.501030 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.500994    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb" event={"ID":"0656b83a-c10d-4929-b029-c3852202b703","Type":"ContainerStarted","Data":"dec47ffb64038d076330765c568a8f6a30b5137099e3b9cab444f477e7d04566"}
Nov 02 02:00:58.501215 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.501061    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb"
Nov 02 02:00:58.501215 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.501083    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb" event={"ID":"0656b83a-c10d-4929-b029-c3852202b703","Type":"ContainerStarted","Data":"4bb6e3293299a2c76e226551470240fd588f53b56d14bacded6303a8a8ebf7eb"}
Nov 02 02:00:58.502844 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.502815    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz" event={"ID":"0d212576-5a9c-4523-8a0f-93b851be6406","Type":"ContainerStarted","Data":"435867d96f9fd47fd0e56cc38fd18a366d60441601ae422ad4a9388fd2327a7d"}
Nov 02 02:00:58.502972 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.502865    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz" event={"ID":"0d212576-5a9c-4523-8a0f-93b851be6406","Type":"ContainerStarted","Data":"f29c1c61c860109e943dc8244c253f8770f3bc76312348d24d6acaf71bd07eb8"}
Nov 02 02:00:58.502972 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.502904    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz"
Nov 02 02:00:58.682067 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.681999    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6"]
Nov 02 02:00:58.687135 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.687109    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6"
Nov 02 02:00:58.740523 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.740464    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5700d3d1-serving-cert\""
Nov 02 02:00:58.746772 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.746748    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6"]
Nov 02 02:00:58.773393 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.773363    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/731b8642-8f48-42d2-82b1-10bece738f1b-maas-api-tls\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6\" (UID: \"731b8642-8f48-42d2-82b1-10bece738f1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6"
Nov 02 02:00:58.773595 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.773573    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/731b8642-8f48-42d2-82b1-10bece738f1b-metrics-certs\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6\" (UID: \"731b8642-8f48-42d2-82b1-10bece738f1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6"
Nov 02 02:00:58.773772 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.773749    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jvlw\" (UniqueName: \"kubernetes.io/projected/731b8642-8f48-42d2-82b1-10bece738f1b-kube-api-access-9jvlw\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6\" (UID: \"731b8642-8f48-42d2-82b1-10bece738f1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6"
Nov 02 02:00:58.773946 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.773923    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/731b8642-8f48-42d2-82b1-10bece738f1b-openshift-service-ca\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6\" (UID: \"731b8642-8f48-42d2-82b1-10bece738f1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6"
Nov 02 02:00:58.852917 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.852867    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz" podStartSLOduration=2.852853312 podStartE2EDuration="2.852853312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:58.850255539 +0000 UTC m=+1706.291292040" watchObservedRunningTime="2026-11-02 02:00:58.852853312 +0000 UTC m=+1706.293889809"
Nov 02 02:00:58.875352 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.875320    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jvlw\" (UniqueName: \"kubernetes.io/projected/731b8642-8f48-42d2-82b1-10bece738f1b-kube-api-access-9jvlw\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6\" (UID: \"731b8642-8f48-42d2-82b1-10bece738f1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6"
Nov 02 02:00:58.875505 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.875471    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/731b8642-8f48-42d2-82b1-10bece738f1b-openshift-service-ca\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6\" (UID: \"731b8642-8f48-42d2-82b1-10bece738f1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6"
Nov 02 02:00:58.875655 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.875639    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/731b8642-8f48-42d2-82b1-10bece738f1b-maas-api-tls\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6\" (UID: \"731b8642-8f48-42d2-82b1-10bece738f1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6"
Nov 02 02:00:58.875749 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.875736    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/731b8642-8f48-42d2-82b1-10bece738f1b-metrics-certs\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6\" (UID: \"731b8642-8f48-42d2-82b1-10bece738f1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6"
Nov 02 02:00:58.875818 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:00:58.875753    5875 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-serving-cert: secret "maas-api-e2e-mt-5700d3d1-serving-cert" not found
Nov 02 02:00:58.875863 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:00:58.875818    5875 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/731b8642-8f48-42d2-82b1-10bece738f1b-maas-api-tls podName:731b8642-8f48-42d2-82b1-10bece738f1b nodeName:}" failed. No retries permitted until 2026-11-02 02:00:59.375797651 +0000 UTC m=+1706.816834126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/731b8642-8f48-42d2-82b1-10bece738f1b-maas-api-tls") pod "maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6" (UID: "731b8642-8f48-42d2-82b1-10bece738f1b") : secret "maas-api-e2e-mt-5700d3d1-serving-cert" not found
Nov 02 02:00:58.876044 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.876026    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/731b8642-8f48-42d2-82b1-10bece738f1b-openshift-service-ca\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6\" (UID: \"731b8642-8f48-42d2-82b1-10bece738f1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6"
Nov 02 02:00:58.878137 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.878115    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/731b8642-8f48-42d2-82b1-10bece738f1b-metrics-certs\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6\" (UID: \"731b8642-8f48-42d2-82b1-10bece738f1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6"
Nov 02 02:00:58.936343 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.936288    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jvlw\" (UniqueName: \"kubernetes.io/projected/731b8642-8f48-42d2-82b1-10bece738f1b-kube-api-access-9jvlw\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6\" (UID: \"731b8642-8f48-42d2-82b1-10bece738f1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6"
Nov 02 02:00:58.981247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:58.973562    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb" podStartSLOduration=2.97354115 podStartE2EDuration="2.97354115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:58.967622074 +0000 UTC m=+1706.408658572" watchObservedRunningTime="2026-11-02 02:00:58.97354115 +0000 UTC m=+1706.414577652"
Nov 02 02:00:59.392075 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:59.392033    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/731b8642-8f48-42d2-82b1-10bece738f1b-maas-api-tls\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6\" (UID: \"731b8642-8f48-42d2-82b1-10bece738f1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6"
Nov 02 02:00:59.394668 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:59.394647    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/731b8642-8f48-42d2-82b1-10bece738f1b-maas-api-tls\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6\" (UID: \"731b8642-8f48-42d2-82b1-10bece738f1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6"
Nov 02 02:00:59.607701 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:59.607664    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6"
Nov 02 02:00:59.882257 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:59.882203    5875 patch_prober.go:29] interesting pod/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:00:59.882419 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:00:59.882276    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7" podUID="3726ebef-c581-4736-b0a3-ae417a6e8414" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:01:00.026278 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.021998    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6"]
Nov 02 02:01:00.050440 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:01:00.047865    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731b8642_8f48_42d2_82b1_10bece738f1b.slice/crio-c6826bda967e9354cb3871fb646ed8d821d06f13e4227c4f171bd73886156119 WatchSource:0}: Error finding container c6826bda967e9354cb3871fb646ed8d821d06f13e4227c4f171bd73886156119: Status 404 returned error can't find the container with id c6826bda967e9354cb3871fb646ed8d821d06f13e4227c4f171bd73886156119
Nov 02 02:01:00.098401 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.098356    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"]
Nov 02 02:01:00.106379 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.105931    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.133694 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.132852    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-workload-socket\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.133694 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.132965    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvmrw\" (UniqueName: \"kubernetes.io/projected/4650bde4-c321-40aa-bc8e-af7bc0dd3311-kube-api-access-lvmrw\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.133694 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.132996    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-token\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.133694 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.133045    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-podinfo\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.133694 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.133087    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-workload-certs\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.133694 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.133181    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-envoy\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.133694 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.133267    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-credential-socket\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.133694 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.133301    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-data\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.133694 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.133402    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istiod-ca-cert\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.211244 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.206748    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-11f26c-openshift-default-dockercfg-fpvnp\""
Nov 02 02:01:00.256393 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.242072    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-podinfo\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.256393 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.242124    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-workload-certs\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.256393 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.242184    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-envoy\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.256393 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.242254    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-credential-socket\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.256393 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.242287    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-data\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.256393 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.242330    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istiod-ca-cert\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.256393 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.242434    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-workload-socket\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.256393 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.242496    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvmrw\" (UniqueName: \"kubernetes.io/projected/4650bde4-c321-40aa-bc8e-af7bc0dd3311-kube-api-access-lvmrw\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.256393 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.242513    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-token\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.263109 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.258876    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-data\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.263109 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.259198    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-workload-certs\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.272437 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.270632    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-workload-socket\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.272437 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.271697    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istiod-ca-cert\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.272437 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.271972    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-credential-socket\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.369727 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.348477    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"]
Nov 02 02:01:00.374180 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.373197    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-token\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.376741 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.376700    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-envoy\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.391251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.387849    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-podinfo\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.406251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.395624    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvmrw\" (UniqueName: \"kubernetes.io/projected/4650bde4-c321-40aa-bc8e-af7bc0dd3311-kube-api-access-lvmrw\") pod \"e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") " pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.523461 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.523363    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:00.672252 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.669408    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:01:00.672252 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.669472    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw" podUID="bceaf892-6e4e-4472-bf8d-34b1fe161036" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:01:00.790793 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:00.783985    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6" event={"ID":"731b8642-8f48-42d2-82b1-10bece738f1b","Type":"ContainerStarted","Data":"c6826bda967e9354cb3871fb646ed8d821d06f13e4227c4f171bd73886156119"}
Nov 02 02:01:01.283693 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:01:01.283665    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4650bde4_c321_40aa_bc8e_af7bc0dd3311.slice/crio-d0705bb93989569a26653bb3773a6efef9d1441ef3a942f833ef0f852a2243fa WatchSource:0}: Error finding container d0705bb93989569a26653bb3773a6efef9d1441ef3a942f833ef0f852a2243fa: Status 404 returned error can't find the container with id d0705bb93989569a26653bb3773a6efef9d1441ef3a942f833ef0f852a2243fa
Nov 02 02:01:01.284258 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.284184    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"]
Nov 02 02:01:01.286802 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.286766    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:01:01.286920 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.286845    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:01:01.286920 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.286891    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:01:01.638573 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.638543    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:01:01.639082 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.639039    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b" podUID="7bef4b3e-a32d-4171-bd35-d19e9feef1e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:01:01.705705 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.705659    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc"]
Nov 02 02:01:01.711770 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.711745    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc"
Nov 02 02:01:01.766652 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.766625    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc"]
Nov 02 02:01:01.787457 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.785719    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:01:01.787457 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.785854    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:01:01.807946 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.806979    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6" event={"ID":"731b8642-8f48-42d2-82b1-10bece738f1b","Type":"ContainerStarted","Data":"9b7e70fc9fca734c913730e76c5a99fce5cbda3078e00530ddcf2b8d3cf54114"}
Nov 02 02:01:01.807946 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.807668    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6"
Nov 02 02:01:01.818275 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.817252    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6181a5e7-b41f-4a80-a256-a2b609c0f551-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc\" (UID: \"6181a5e7-b41f-4a80-a256-a2b609c0f551\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc"
Nov 02 02:01:01.818275 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.817498    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgsxl\" (UniqueName: \"kubernetes.io/projected/6181a5e7-b41f-4a80-a256-a2b609c0f551-kube-api-access-fgsxl\") pod \"maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc\" (UID: \"6181a5e7-b41f-4a80-a256-a2b609c0f551\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc"
Nov 02 02:01:01.818275 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.817643    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6181a5e7-b41f-4a80-a256-a2b609c0f551-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc\" (UID: \"6181a5e7-b41f-4a80-a256-a2b609c0f551\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc"
Nov 02 02:01:01.819510 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.819253    5875 generic.go:356] "Generic (PLEG): container finished" podID="6b0c4e37-e032-4b48-8261-81294adf8635" containerID="9b8d5f60b15c963b43d64fec47ac742c52678a4f5f9df8c5dcb09f64c73ce6a7" exitCode=0
Nov 02 02:01:01.819510 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.819401    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n" event={"ID":"6b0c4e37-e032-4b48-8261-81294adf8635","Type":"ContainerDied","Data":"9b8d5f60b15c963b43d64fec47ac742c52678a4f5f9df8c5dcb09f64c73ce6a7"}
Nov 02 02:01:01.822330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.822242    5875 generic.go:356] "Generic (PLEG): container finished" podID="05f7e0b6-dc17-4891-9817-242be8465c54" containerID="7dcf8805adcd0c037b18a35a45dfc5dc6a190a503be98f61da4dfe2a49e2891b" exitCode=0
Nov 02 02:01:01.822450 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.822379    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh" event={"ID":"05f7e0b6-dc17-4891-9817-242be8465c54","Type":"ContainerDied","Data":"7dcf8805adcd0c037b18a35a45dfc5dc6a190a503be98f61da4dfe2a49e2891b"}
Nov 02 02:01:01.822450 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.822425    5875 scope.go:122] "RemoveContainer" containerID="295e19acbb53caf69ec7f0f20cf688103e7b889e62d320dc8c7e525a17c1f33d"
Nov 02 02:01:01.826261 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.826177    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92" event={"ID":"4650bde4-c321-40aa-bc8e-af7bc0dd3311","Type":"ContainerStarted","Data":"d3e2a3f5397eefe994e1bbd3e203ce3edb73f56cff8cf6f0757f72524a2ffc58"}
Nov 02 02:01:01.826428 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.826394    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92" event={"ID":"4650bde4-c321-40aa-bc8e-af7bc0dd3311","Type":"ContainerStarted","Data":"d0705bb93989569a26653bb3773a6efef9d1441ef3a942f833ef0f852a2243fa"}
Nov 02 02:01:01.839440 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.839284    5875 generic.go:356] "Generic (PLEG): container finished" podID="3f4aa706-493e-42ec-a124-187020d8aa3e" containerID="ec384196885d2de31d686f8701f35bad124fcebbd5b54b9b32f356131a18677e" exitCode=0
Nov 02 02:01:01.839440 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.839388    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f" event={"ID":"3f4aa706-493e-42ec-a124-187020d8aa3e","Type":"ContainerDied","Data":"ec384196885d2de31d686f8701f35bad124fcebbd5b54b9b32f356131a18677e"}
Nov 02 02:01:01.921920 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.919746    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6181a5e7-b41f-4a80-a256-a2b609c0f551-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc\" (UID: \"6181a5e7-b41f-4a80-a256-a2b609c0f551\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc"
Nov 02 02:01:01.921920 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.920001    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6181a5e7-b41f-4a80-a256-a2b609c0f551-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc\" (UID: \"6181a5e7-b41f-4a80-a256-a2b609c0f551\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc"
Nov 02 02:01:01.921920 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.920213    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgsxl\" (UniqueName: \"kubernetes.io/projected/6181a5e7-b41f-4a80-a256-a2b609c0f551-kube-api-access-fgsxl\") pod \"maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc\" (UID: \"6181a5e7-b41f-4a80-a256-a2b609c0f551\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc"
Nov 02 02:01:01.922204 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.922185    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6181a5e7-b41f-4a80-a256-a2b609c0f551-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc\" (UID: \"6181a5e7-b41f-4a80-a256-a2b609c0f551\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc"
Nov 02 02:01:01.924481 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.924457    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6181a5e7-b41f-4a80-a256-a2b609c0f551-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc\" (UID: \"6181a5e7-b41f-4a80-a256-a2b609c0f551\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc"
Nov 02 02:01:01.953566 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.950515    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6" podStartSLOduration=2.816696144 podStartE2EDuration="3.950495062s" totalImagesPullingTime="1.133798918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:01:01.948752088 +0000 UTC m=+1709.389788598" watchObservedRunningTime="2026-11-02 02:01:01.950495062 +0000 UTC m=+1709.391531560"
Nov 02 02:01:01.959676 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.959592    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 02:01:01.984244 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:01.984191    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgsxl\" (UniqueName: \"kubernetes.io/projected/6181a5e7-b41f-4a80-a256-a2b609c0f551-kube-api-access-fgsxl\") pod \"maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc\" (UID: \"6181a5e7-b41f-4a80-a256-a2b609c0f551\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc"
Nov 02 02:01:02.014384 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.014365    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 02:01:02.018086 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.018068    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 02:01:02.021546 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.021518    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b0c4e37-e032-4b48-8261-81294adf8635-istio-token\" (UniqueName: \"kubernetes.io/projected/6b0c4e37-e032-4b48-8261-81294adf8635-istio-token\") pod \"6b0c4e37-e032-4b48-8261-81294adf8635\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") "
Nov 02 02:01:02.021609 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.021558    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-istio-data\") pod \"6b0c4e37-e032-4b48-8261-81294adf8635\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") "
Nov 02 02:01:02.021666 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.021610    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-istio-envoy\") pod \"6b0c4e37-e032-4b48-8261-81294adf8635\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") "
Nov 02 02:01:02.021719 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.021662    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-workload-socket\") pod \"6b0c4e37-e032-4b48-8261-81294adf8635\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") "
Nov 02 02:01:02.021831 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.021813    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b0c4e37-e032-4b48-8261-81294adf8635-kube-api-access-ktzqf\" (UniqueName: \"kubernetes.io/projected/6b0c4e37-e032-4b48-8261-81294adf8635-kube-api-access-ktzqf\") pod \"6b0c4e37-e032-4b48-8261-81294adf8635\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") "
Nov 02 02:01:02.021921 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.021901    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-istio-data" pod "6b0c4e37-e032-4b48-8261-81294adf8635" (UID: "6b0c4e37-e032-4b48-8261-81294adf8635"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:02.021964 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.021914    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-workload-socket" pod "6b0c4e37-e032-4b48-8261-81294adf8635" (UID: "6b0c4e37-e032-4b48-8261-81294adf8635"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:02.022145 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.022130    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-credential-socket\") pod \"6b0c4e37-e032-4b48-8261-81294adf8635\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") "
Nov 02 02:01:02.022263 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.022169    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b0c4e37-e032-4b48-8261-81294adf8635-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b0c4e37-e032-4b48-8261-81294adf8635-istiod-ca-cert\") pod \"6b0c4e37-e032-4b48-8261-81294adf8635\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") "
Nov 02 02:01:02.022263 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.022246    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b0c4e37-e032-4b48-8261-81294adf8635-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b0c4e37-e032-4b48-8261-81294adf8635-istio-podinfo\") pod \"6b0c4e37-e032-4b48-8261-81294adf8635\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") "
Nov 02 02:01:02.022422 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.022365    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-workload-certs\") pod \"6b0c4e37-e032-4b48-8261-81294adf8635\" (UID: \"6b0c4e37-e032-4b48-8261-81294adf8635\") "
Nov 02 02:01:02.022600 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.022440    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-credential-socket" pod "6b0c4e37-e032-4b48-8261-81294adf8635" (UID: "6b0c4e37-e032-4b48-8261-81294adf8635"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:02.022735 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.022709    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-workload-certs" pod "6b0c4e37-e032-4b48-8261-81294adf8635" (UID: "6b0c4e37-e032-4b48-8261-81294adf8635"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:02.022816 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.022793    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b0c4e37-e032-4b48-8261-81294adf8635-istiod-ca-cert" pod "6b0c4e37-e032-4b48-8261-81294adf8635" (UID: "6b0c4e37-e032-4b48-8261-81294adf8635"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:02.023507 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.023480    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.023611 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.023513    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.023611 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.023529    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.023611 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.023543    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.023611 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.023558    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b0c4e37-e032-4b48-8261-81294adf8635-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.023862 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.023834    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b0c4e37-e032-4b48-8261-81294adf8635-istio-token" pod "6b0c4e37-e032-4b48-8261-81294adf8635" (UID: "6b0c4e37-e032-4b48-8261-81294adf8635"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:02.024110 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.024088    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-istio-envoy" pod "6b0c4e37-e032-4b48-8261-81294adf8635" (UID: "6b0c4e37-e032-4b48-8261-81294adf8635"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:02.024372 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.024348    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b0c4e37-e032-4b48-8261-81294adf8635-kube-api-access-ktzqf" pod "6b0c4e37-e032-4b48-8261-81294adf8635" (UID: "6b0c4e37-e032-4b48-8261-81294adf8635"). InnerVolumeSpecName "kube-api-access-ktzqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:02.024644 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.024622    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b0c4e37-e032-4b48-8261-81294adf8635-istio-podinfo" pod "6b0c4e37-e032-4b48-8261-81294adf8635" (UID: "6b0c4e37-e032-4b48-8261-81294adf8635"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:01:02.145795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.143326    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f7e0b6-dc17-4891-9817-242be8465c54-istio-token\" (UniqueName: \"kubernetes.io/projected/05f7e0b6-dc17-4891-9817-242be8465c54-istio-token\") pod \"05f7e0b6-dc17-4891-9817-242be8465c54\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") "
Nov 02 02:01:02.145795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.143377    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f7e0b6-dc17-4891-9817-242be8465c54-kube-api-access-kjczf\" (UniqueName: \"kubernetes.io/projected/05f7e0b6-dc17-4891-9817-242be8465c54-kube-api-access-kjczf\") pod \"05f7e0b6-dc17-4891-9817-242be8465c54\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") "
Nov 02 02:01:02.145795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.143443    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-istio-envoy\") pod \"05f7e0b6-dc17-4891-9817-242be8465c54\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") "
Nov 02 02:01:02.145795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.143508    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-workload-certs\") pod \"05f7e0b6-dc17-4891-9817-242be8465c54\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") "
Nov 02 02:01:02.145795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.143564    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-workload-socket\") pod \"05f7e0b6-dc17-4891-9817-242be8465c54\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") "
Nov 02 02:01:02.145795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.143651    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-envoy\") pod \"3f4aa706-493e-42ec-a124-187020d8aa3e\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") "
Nov 02 02:01:02.145795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.143697    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-workload-socket\") pod \"3f4aa706-493e-42ec-a124-187020d8aa3e\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") "
Nov 02 02:01:02.145795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.143800    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-data\") pod \"3f4aa706-493e-42ec-a124-187020d8aa3e\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") "
Nov 02 02:01:02.145795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.143851    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f4aa706-493e-42ec-a124-187020d8aa3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4aa706-493e-42ec-a124-187020d8aa3e-istiod-ca-cert\") pod \"3f4aa706-493e-42ec-a124-187020d8aa3e\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") "
Nov 02 02:01:02.145795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.143877    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f7e0b6-dc17-4891-9817-242be8465c54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f7e0b6-dc17-4891-9817-242be8465c54-istiod-ca-cert\") pod \"05f7e0b6-dc17-4891-9817-242be8465c54\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") "
Nov 02 02:01:02.145795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.143938    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-token\" (UniqueName: \"kubernetes.io/projected/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-token\") pod \"3f4aa706-493e-42ec-a124-187020d8aa3e\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") "
Nov 02 02:01:02.145795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.143992    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05f7e0b6-dc17-4891-9817-242be8465c54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f7e0b6-dc17-4891-9817-242be8465c54-istio-podinfo\") pod \"05f7e0b6-dc17-4891-9817-242be8465c54\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") "
Nov 02 02:01:02.145795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.144018    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4aa706-493e-42ec-a124-187020d8aa3e-kube-api-access-7f668\" (UniqueName: \"kubernetes.io/projected/3f4aa706-493e-42ec-a124-187020d8aa3e-kube-api-access-7f668\") pod \"3f4aa706-493e-42ec-a124-187020d8aa3e\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") "
Nov 02 02:01:02.145795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.144063    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-credential-socket\") pod \"05f7e0b6-dc17-4891-9817-242be8465c54\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") "
Nov 02 02:01:02.145795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.144100    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-istio-data\") pod \"05f7e0b6-dc17-4891-9817-242be8465c54\" (UID: \"05f7e0b6-dc17-4891-9817-242be8465c54\") "
Nov 02 02:01:02.145795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.144130    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-credential-socket\") pod \"3f4aa706-493e-42ec-a124-187020d8aa3e\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") "
Nov 02 02:01:02.145795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.144242    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-podinfo\") pod \"3f4aa706-493e-42ec-a124-187020d8aa3e\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") "
Nov 02 02:01:02.145795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.144346    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-workload-certs\") pod \"3f4aa706-493e-42ec-a124-187020d8aa3e\" (UID: \"3f4aa706-493e-42ec-a124-187020d8aa3e\") "
Nov 02 02:01:02.145795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.145160    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktzqf\" (UniqueName: \"kubernetes.io/projected/6b0c4e37-e032-4b48-8261-81294adf8635-kube-api-access-ktzqf\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.145795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.145182    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b0c4e37-e032-4b48-8261-81294adf8635-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.145795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.145196    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b0c4e37-e032-4b48-8261-81294adf8635-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.145795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.145210    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b0c4e37-e032-4b48-8261-81294adf8635-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.145795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.145494    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-workload-certs" pod "3f4aa706-493e-42ec-a124-187020d8aa3e" (UID: "3f4aa706-493e-42ec-a124-187020d8aa3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:02.164961 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.164478    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-workload-certs" pod "05f7e0b6-dc17-4891-9817-242be8465c54" (UID: "05f7e0b6-dc17-4891-9817-242be8465c54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:02.164961 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.164688    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-workload-socket" pod "05f7e0b6-dc17-4891-9817-242be8465c54" (UID: "05f7e0b6-dc17-4891-9817-242be8465c54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:02.184797 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.170700    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f7e0b6-dc17-4891-9817-242be8465c54-istio-token" pod "05f7e0b6-dc17-4891-9817-242be8465c54" (UID: "05f7e0b6-dc17-4891-9817-242be8465c54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:02.184797 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.173174    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-istio-envoy" pod "05f7e0b6-dc17-4891-9817-242be8465c54" (UID: "05f7e0b6-dc17-4891-9817-242be8465c54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:02.184797 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.173550    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f4aa706-493e-42ec-a124-187020d8aa3e-istiod-ca-cert" pod "3f4aa706-493e-42ec-a124-187020d8aa3e" (UID: "3f4aa706-493e-42ec-a124-187020d8aa3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:02.184797 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.173771    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-workload-socket" pod "3f4aa706-493e-42ec-a124-187020d8aa3e" (UID: "3f4aa706-493e-42ec-a124-187020d8aa3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:02.184797 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.173925    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-data" pod "3f4aa706-493e-42ec-a124-187020d8aa3e" (UID: "3f4aa706-493e-42ec-a124-187020d8aa3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:02.184797 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.174080    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-istio-data" pod "05f7e0b6-dc17-4891-9817-242be8465c54" (UID: "05f7e0b6-dc17-4891-9817-242be8465c54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:02.184797 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.178771    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f7e0b6-dc17-4891-9817-242be8465c54-kube-api-access-kjczf" pod "05f7e0b6-dc17-4891-9817-242be8465c54" (UID: "05f7e0b6-dc17-4891-9817-242be8465c54"). InnerVolumeSpecName "kube-api-access-kjczf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:02.184797 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.184483    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-credential-socket" pod "3f4aa706-493e-42ec-a124-187020d8aa3e" (UID: "3f4aa706-493e-42ec-a124-187020d8aa3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:02.186421 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.186389    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-credential-socket" pod "05f7e0b6-dc17-4891-9817-242be8465c54" (UID: "05f7e0b6-dc17-4891-9817-242be8465c54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:02.186535 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.186467    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-envoy" pod "3f4aa706-493e-42ec-a124-187020d8aa3e" (UID: "3f4aa706-493e-42ec-a124-187020d8aa3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:02.201869 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.201824    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f7e0b6-dc17-4891-9817-242be8465c54-istiod-ca-cert" pod "05f7e0b6-dc17-4891-9817-242be8465c54" (UID: "05f7e0b6-dc17-4891-9817-242be8465c54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:02.229245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.214468    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc"
Nov 02 02:01:02.262565 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.248001    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05f7e0b6-dc17-4891-9817-242be8465c54-istio-podinfo" pod "05f7e0b6-dc17-4891-9817-242be8465c54" (UID: "05f7e0b6-dc17-4891-9817-242be8465c54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:01:02.262565 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.248942    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4aa706-493e-42ec-a124-187020d8aa3e-kube-api-access-7f668" pod "3f4aa706-493e-42ec-a124-187020d8aa3e" (UID: "3f4aa706-493e-42ec-a124-187020d8aa3e"). InnerVolumeSpecName "kube-api-access-7f668". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:02.262565 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.252352    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-podinfo" pod "3f4aa706-493e-42ec-a124-187020d8aa3e" (UID: "3f4aa706-493e-42ec-a124-187020d8aa3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:01:02.326104 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.281569    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92" podStartSLOduration=2.2815433 podStartE2EDuration="2.2815433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:01:02.142597462 +0000 UTC m=+1709.583633960" watchObservedRunningTime="2026-11-02 02:01:02.2815433 +0000 UTC m=+1709.722579798"
Nov 02 02:01:02.326104 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.287309    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-token" pod "3f4aa706-493e-42ec-a124-187020d8aa3e" (UID: "3f4aa706-493e-42ec-a124-187020d8aa3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:02.326104 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.306713    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.326104 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.306745    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.326104 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.306761    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f7e0b6-dc17-4891-9817-242be8465c54-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.326104 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.306778    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjczf\" (UniqueName: \"kubernetes.io/projected/05f7e0b6-dc17-4891-9817-242be8465c54-kube-api-access-kjczf\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.326104 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.306795    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.326104 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.306810    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.326104 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.306823    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.326104 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.306906    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.326104 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.306971    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.326104 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.306992    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.326104 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.307014    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4aa706-493e-42ec-a124-187020d8aa3e-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.326104 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.307037    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f7e0b6-dc17-4891-9817-242be8465c54-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.326104 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.307052    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4aa706-493e-42ec-a124-187020d8aa3e-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.326104 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.307066    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f7e0b6-dc17-4891-9817-242be8465c54-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.326104 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.307082    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f668\" (UniqueName: \"kubernetes.io/projected/3f4aa706-493e-42ec-a124-187020d8aa3e-kube-api-access-7f668\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.326104 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.307097    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.326104 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.307112    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f7e0b6-dc17-4891-9817-242be8465c54-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.326104 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.307128    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4aa706-493e-42ec-a124-187020d8aa3e-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.524637 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.524559    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:02.530056 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.530032    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:02.626825 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.626798    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 02:01:02.724686 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.724629    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-workload-certs\") pod \"3726ebef-c581-4736-b0a3-ae417a6e8414\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") "
Nov 02 02:01:02.725046 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.724713    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-envoy\") pod \"3726ebef-c581-4736-b0a3-ae417a6e8414\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") "
Nov 02 02:01:02.725046 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.724760    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3726ebef-c581-4736-b0a3-ae417a6e8414-kube-api-access-xzqc7\" (UniqueName: \"kubernetes.io/projected/3726ebef-c581-4736-b0a3-ae417a6e8414-kube-api-access-xzqc7\") pod \"3726ebef-c581-4736-b0a3-ae417a6e8414\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") "
Nov 02 02:01:02.725046 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.724864    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3726ebef-c581-4736-b0a3-ae417a6e8414-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3726ebef-c581-4736-b0a3-ae417a6e8414-istiod-ca-cert\") pod \"3726ebef-c581-4736-b0a3-ae417a6e8414\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") "
Nov 02 02:01:02.725046 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.724884    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-workload-certs" pod "3726ebef-c581-4736-b0a3-ae417a6e8414" (UID: "3726ebef-c581-4736-b0a3-ae417a6e8414"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:02.725046 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.724932    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-data\") pod \"3726ebef-c581-4736-b0a3-ae417a6e8414\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") "
Nov 02 02:01:02.725046 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.724971    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-workload-socket\") pod \"3726ebef-c581-4736-b0a3-ae417a6e8414\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") "
Nov 02 02:01:02.725046 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.725039    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-credential-socket\") pod \"3726ebef-c581-4736-b0a3-ae417a6e8414\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") "
Nov 02 02:01:02.726110 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.725115    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-podinfo\") pod \"3726ebef-c581-4736-b0a3-ae417a6e8414\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") "
Nov 02 02:01:02.726110 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.725181    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-token\" (UniqueName: \"kubernetes.io/projected/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-token\") pod \"3726ebef-c581-4736-b0a3-ae417a6e8414\" (UID: \"3726ebef-c581-4736-b0a3-ae417a6e8414\") "
Nov 02 02:01:02.726110 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.725247    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-data" pod "3726ebef-c581-4736-b0a3-ae417a6e8414" (UID: "3726ebef-c581-4736-b0a3-ae417a6e8414"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:02.726110 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.725360    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-workload-socket" pod "3726ebef-c581-4736-b0a3-ae417a6e8414" (UID: "3726ebef-c581-4736-b0a3-ae417a6e8414"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:02.726110 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.725476    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-credential-socket" pod "3726ebef-c581-4736-b0a3-ae417a6e8414" (UID: "3726ebef-c581-4736-b0a3-ae417a6e8414"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:02.726110 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.725845    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3726ebef-c581-4736-b0a3-ae417a6e8414-istiod-ca-cert" pod "3726ebef-c581-4736-b0a3-ae417a6e8414" (UID: "3726ebef-c581-4736-b0a3-ae417a6e8414"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:02.726456 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.726415    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3726ebef-c581-4736-b0a3-ae417a6e8414-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.726456 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.726442    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.726563 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.726459    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.726563 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.726474    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.726563 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.726487    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.727978 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.727947    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-podinfo" pod "3726ebef-c581-4736-b0a3-ae417a6e8414" (UID: "3726ebef-c581-4736-b0a3-ae417a6e8414"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:01:02.728576 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.728556    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-token" pod "3726ebef-c581-4736-b0a3-ae417a6e8414" (UID: "3726ebef-c581-4736-b0a3-ae417a6e8414"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:02.735607 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.735587    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-envoy" pod "3726ebef-c581-4736-b0a3-ae417a6e8414" (UID: "3726ebef-c581-4736-b0a3-ae417a6e8414"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:02.744555 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.744527    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3726ebef-c581-4736-b0a3-ae417a6e8414-kube-api-access-xzqc7" pod "3726ebef-c581-4736-b0a3-ae417a6e8414" (UID: "3726ebef-c581-4736-b0a3-ae417a6e8414"). InnerVolumeSpecName "kube-api-access-xzqc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:02.797824 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.797798    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc"]
Nov 02 02:01:02.812110 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:01:02.812083    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6181a5e7_b41f_4a80_a256_a2b609c0f551.slice/crio-52758232c521b739aa37142e78f9b2118b25eacf34ed355f9011f533e47445fd WatchSource:0}: Error finding container 52758232c521b739aa37142e78f9b2118b25eacf34ed355f9011f533e47445fd: Status 404 returned error can't find the container with id 52758232c521b739aa37142e78f9b2118b25eacf34ed355f9011f533e47445fd
Nov 02 02:01:02.828201 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.828173    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.828329 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.828207    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzqc7\" (UniqueName: \"kubernetes.io/projected/3726ebef-c581-4736-b0a3-ae417a6e8414-kube-api-access-xzqc7\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.828329 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.828240    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.828329 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.828253    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3726ebef-c581-4736-b0a3-ae417a6e8414-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:02.851128 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.851100    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc" event={"ID":"6181a5e7-b41f-4a80-a256-a2b609c0f551","Type":"ContainerStarted","Data":"52758232c521b739aa37142e78f9b2118b25eacf34ed355f9011f533e47445fd"}
Nov 02 02:01:02.852689 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.852670    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"
Nov 02 02:01:02.852689 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.852678    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n" event={"ID":"6b0c4e37-e032-4b48-8261-81294adf8635","Type":"ContainerDied","Data":"cc5bfd73b9c3304f874932781bfef18f7833d026c427aaf84549e89df8c59864"}
Nov 02 02:01:02.852919 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.852738    5875 scope.go:122] "RemoveContainer" containerID="9b8d5f60b15c963b43d64fec47ac742c52678a4f5f9df8c5dcb09f64c73ce6a7"
Nov 02 02:01:02.854611 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.854592    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"
Nov 02 02:01:02.854727 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.854636    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh" event={"ID":"05f7e0b6-dc17-4891-9817-242be8465c54","Type":"ContainerDied","Data":"8b4eca52e36194a9c62fe1939f70a6e20ab14aadd4cbc499ad28d3a0e493bc60"}
Nov 02 02:01:02.856603 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.856577    5875 generic.go:356] "Generic (PLEG): container finished" podID="3726ebef-c581-4736-b0a3-ae417a6e8414" containerID="90a1419c7d2b6bbe194675665acef20037e9d57e1e1f3be8fee60b2d03ec4d49" exitCode=0
Nov 02 02:01:02.856715 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.856639    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"
Nov 02 02:01:02.856792 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.856749    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7" event={"ID":"3726ebef-c581-4736-b0a3-ae417a6e8414","Type":"ContainerDied","Data":"90a1419c7d2b6bbe194675665acef20037e9d57e1e1f3be8fee60b2d03ec4d49"}
Nov 02 02:01:02.856849 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.856792    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7" event={"ID":"3726ebef-c581-4736-b0a3-ae417a6e8414","Type":"ContainerDied","Data":"f0781d30926f6606d69f840ebaf63339f2e1ac1ccae563492bcf8bbb3907ce9d"}
Nov 02 02:01:02.860846 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.860789    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f" event={"ID":"3f4aa706-493e-42ec-a124-187020d8aa3e","Type":"ContainerDied","Data":"de15774b6381e904d48e201dbc290cb1d7a1e29e37655f1aeec6d64dc28f7650"}
Nov 02 02:01:02.860846 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.860833    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"
Nov 02 02:01:02.861185 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.861138    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:02.866315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.866295    5875 scope.go:122] "RemoveContainer" containerID="7dcf8805adcd0c037b18a35a45dfc5dc6a190a503be98f61da4dfe2a49e2891b"
Nov 02 02:01:02.867076 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.867021    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:01:02.877273 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.877248    5875 scope.go:122] "RemoveContainer" containerID="90a1419c7d2b6bbe194675665acef20037e9d57e1e1f3be8fee60b2d03ec4d49"
Nov 02 02:01:02.886931 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.886912    5875 scope.go:122] "RemoveContainer" containerID="90a1419c7d2b6bbe194675665acef20037e9d57e1e1f3be8fee60b2d03ec4d49"
Nov 02 02:01:02.887215 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:01:02.887194    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90a1419c7d2b6bbe194675665acef20037e9d57e1e1f3be8fee60b2d03ec4d49\": container with ID starting with 90a1419c7d2b6bbe194675665acef20037e9d57e1e1f3be8fee60b2d03ec4d49 not found: ID does not exist" containerID="90a1419c7d2b6bbe194675665acef20037e9d57e1e1f3be8fee60b2d03ec4d49"
Nov 02 02:01:02.887305 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.887250    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90a1419c7d2b6bbe194675665acef20037e9d57e1e1f3be8fee60b2d03ec4d49"} err="failed to get container status \"90a1419c7d2b6bbe194675665acef20037e9d57e1e1f3be8fee60b2d03ec4d49\": rpc error: code = NotFound desc = could not find container \"90a1419c7d2b6bbe194675665acef20037e9d57e1e1f3be8fee60b2d03ec4d49\": container with ID starting with 90a1419c7d2b6bbe194675665acef20037e9d57e1e1f3be8fee60b2d03ec4d49 not found: ID does not exist"
Nov 02 02:01:02.887305 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.887270    5875 scope.go:122] "RemoveContainer" containerID="ec384196885d2de31d686f8701f35bad124fcebbd5b54b9b32f356131a18677e"
Nov 02 02:01:02.939662 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.939623    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:01:02.939830 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:02.939669    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k" podUID="5ec9fbc9-a60c-4a82-932c-0af50b66ecab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:01:03.074728 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.074645    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"]
Nov 02 02:01:03.078571 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.078550    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-bj57n"]
Nov 02 02:01:03.187951 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.187918    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"]
Nov 02 02:01:03.192471 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.192444    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-qcphh"]
Nov 02 02:01:03.474440 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.474355    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"]
Nov 02 02:01:03.508439 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.505736    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-x2x4f"]
Nov 02 02:01:03.583277 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.583181    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"]
Nov 02 02:01:03.589310 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.589274    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ebdb3-openshift-default-769ffb6bd5-js4b7"]
Nov 02 02:01:03.709109 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.709079    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 02:01:03.778075 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.778041    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-data\") pod \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") "
Nov 02 02:01:03.778561 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.778154    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istiod-ca-cert\") pod \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") "
Nov 02 02:01:03.778561 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.778207    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-podinfo\") pod \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") "
Nov 02 02:01:03.778561 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.778304    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-credential-socket\") pod \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") "
Nov 02 02:01:03.778561 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.778350    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-kube-api-access-xrzj5\" (UniqueName: \"kubernetes.io/projected/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-kube-api-access-xrzj5\") pod \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") "
Nov 02 02:01:03.778561 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.778362    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-data" pod "7bef4b3e-a32d-4171-bd35-d19e9feef1e4" (UID: "7bef4b3e-a32d-4171-bd35-d19e9feef1e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:03.778561 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.778503    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-workload-socket\") pod \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") "
Nov 02 02:01:03.778990 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.778574    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-envoy\") pod \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") "
Nov 02 02:01:03.778990 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.778577    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-credential-socket" pod "7bef4b3e-a32d-4171-bd35-d19e9feef1e4" (UID: "7bef4b3e-a32d-4171-bd35-d19e9feef1e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:03.778990 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.778619    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istiod-ca-cert" pod "7bef4b3e-a32d-4171-bd35-d19e9feef1e4" (UID: "7bef4b3e-a32d-4171-bd35-d19e9feef1e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:03.778990 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.778688    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-workload-certs\") pod \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") "
Nov 02 02:01:03.778990 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.778718    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-token\" (UniqueName: \"kubernetes.io/projected/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-token\") pod \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\" (UID: \"7bef4b3e-a32d-4171-bd35-d19e9feef1e4\") "
Nov 02 02:01:03.778990 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.778800    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-workload-socket" pod "7bef4b3e-a32d-4171-bd35-d19e9feef1e4" (UID: "7bef4b3e-a32d-4171-bd35-d19e9feef1e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:03.778990 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.778968    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-workload-certs" pod "7bef4b3e-a32d-4171-bd35-d19e9feef1e4" (UID: "7bef4b3e-a32d-4171-bd35-d19e9feef1e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:03.779616 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.779599    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:03.779749 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.779732    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:03.779816 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.779752    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:03.779816 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.779768    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:03.779816 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.779782    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:03.782239 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.782195    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-envoy" pod "7bef4b3e-a32d-4171-bd35-d19e9feef1e4" (UID: "7bef4b3e-a32d-4171-bd35-d19e9feef1e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:03.783094 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.783075    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-podinfo" pod "7bef4b3e-a32d-4171-bd35-d19e9feef1e4" (UID: "7bef4b3e-a32d-4171-bd35-d19e9feef1e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:01:03.783984 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.783963    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-token" pod "7bef4b3e-a32d-4171-bd35-d19e9feef1e4" (UID: "7bef4b3e-a32d-4171-bd35-d19e9feef1e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:03.784067 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.784032    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-kube-api-access-xrzj5" pod "7bef4b3e-a32d-4171-bd35-d19e9feef1e4" (UID: "7bef4b3e-a32d-4171-bd35-d19e9feef1e4"). InnerVolumeSpecName "kube-api-access-xrzj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:03.827148 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.827127    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 02:01:03.829589 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.829567    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 02:01:03.837352 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.837302    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 02:01:03.874936 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.874889    5875 generic.go:356] "Generic (PLEG): container finished" podID="bceaf892-6e4e-4472-bf8d-34b1fe161036" containerID="867fc8e3aacab21720f28a83a60d87a1999b0b1cb75cf4a6e7c6ae051b997ce7" exitCode=0
Nov 02 02:01:03.875102 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.875015    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"
Nov 02 02:01:03.875102 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.875022    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw" event={"ID":"bceaf892-6e4e-4472-bf8d-34b1fe161036","Type":"ContainerDied","Data":"867fc8e3aacab21720f28a83a60d87a1999b0b1cb75cf4a6e7c6ae051b997ce7"}
Nov 02 02:01:03.875102 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.875079    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw" event={"ID":"bceaf892-6e4e-4472-bf8d-34b1fe161036","Type":"ContainerDied","Data":"a816bcb373c4eebe2a1bd1fea6616627d557e783c60543936143616010a2983b"}
Nov 02 02:01:03.875278 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.875108    5875 scope.go:122] "RemoveContainer" containerID="867fc8e3aacab21720f28a83a60d87a1999b0b1cb75cf4a6e7c6ae051b997ce7"
Nov 02 02:01:03.881157 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.881129    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:03.881292 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.881162    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:03.881292 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.881177    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:03.881292 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.881194    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrzj5\" (UniqueName: \"kubernetes.io/projected/7bef4b3e-a32d-4171-bd35-d19e9feef1e4-kube-api-access-xrzj5\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:03.882215 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.881869    5875 generic.go:356] "Generic (PLEG): container finished" podID="5ec9fbc9-a60c-4a82-932c-0af50b66ecab" containerID="5bbac46e87a02af71b7c6653d29432628bcdf0f3f0ae71c70b984912a2bd30c5" exitCode=0
Nov 02 02:01:03.882215 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.881983    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k" event={"ID":"5ec9fbc9-a60c-4a82-932c-0af50b66ecab","Type":"ContainerDied","Data":"5bbac46e87a02af71b7c6653d29432628bcdf0f3f0ae71c70b984912a2bd30c5"}
Nov 02 02:01:03.882215 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.882020    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k" event={"ID":"5ec9fbc9-a60c-4a82-932c-0af50b66ecab","Type":"ContainerDied","Data":"b6bbeb20c5fbfe3870fbb45a07a045ecbe7eb238b50da5df9036b607cc3d55a5"}
Nov 02 02:01:03.882215 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.882080    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"
Nov 02 02:01:03.884347 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.883970    5875 generic.go:356] "Generic (PLEG): container finished" podID="c637ee2b-9c6b-489e-8f6b-49112217a03c" containerID="ca1bf69d08ce3a5c2eb3ec7b5bfd9869e3244d66b88d50cd5da80b992579494a" exitCode=0
Nov 02 02:01:03.884347 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.884044    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt" event={"ID":"c637ee2b-9c6b-489e-8f6b-49112217a03c","Type":"ContainerDied","Data":"ca1bf69d08ce3a5c2eb3ec7b5bfd9869e3244d66b88d50cd5da80b992579494a"}
Nov 02 02:01:03.884347 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.884081    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt" event={"ID":"c637ee2b-9c6b-489e-8f6b-49112217a03c","Type":"ContainerDied","Data":"542d5ad915a61eb010087dc94e940f4a45c2a8c7989d8371d28225511e5b6f76"}
Nov 02 02:01:03.884347 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.884154    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"
Nov 02 02:01:03.886008 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.885948    5875 generic.go:356] "Generic (PLEG): container finished" podID="7bef4b3e-a32d-4171-bd35-d19e9feef1e4" containerID="1408e3ce8a3aea7a7c66569514dc05028c2af17e9ffcce6b620da594484ce89a" exitCode=0
Nov 02 02:01:03.886008 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.885978    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b" event={"ID":"7bef4b3e-a32d-4171-bd35-d19e9feef1e4","Type":"ContainerDied","Data":"1408e3ce8a3aea7a7c66569514dc05028c2af17e9ffcce6b620da594484ce89a"}
Nov 02 02:01:03.886147 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.886037    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b" event={"ID":"7bef4b3e-a32d-4171-bd35-d19e9feef1e4","Type":"ContainerDied","Data":"c08d83051db7a76614bfd70c7253f1080e56f6c4115544d2a036d94d514aa39a"}
Nov 02 02:01:03.886147 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.885986    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"
Nov 02 02:01:03.887834 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.887809    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc" event={"ID":"6181a5e7-b41f-4a80-a256-a2b609c0f551","Type":"ContainerStarted","Data":"f2d9690b23888988760664c3ad7ac8b0d867a1229a0502d44de05069bb6ff44c"}
Nov 02 02:01:03.920361 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.920343    5875 scope.go:122] "RemoveContainer" containerID="867fc8e3aacab21720f28a83a60d87a1999b0b1cb75cf4a6e7c6ae051b997ce7"
Nov 02 02:01:03.920694 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:01:03.920674    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"867fc8e3aacab21720f28a83a60d87a1999b0b1cb75cf4a6e7c6ae051b997ce7\": container with ID starting with 867fc8e3aacab21720f28a83a60d87a1999b0b1cb75cf4a6e7c6ae051b997ce7 not found: ID does not exist" containerID="867fc8e3aacab21720f28a83a60d87a1999b0b1cb75cf4a6e7c6ae051b997ce7"
Nov 02 02:01:03.920775 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.920705    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"867fc8e3aacab21720f28a83a60d87a1999b0b1cb75cf4a6e7c6ae051b997ce7"} err="failed to get container status \"867fc8e3aacab21720f28a83a60d87a1999b0b1cb75cf4a6e7c6ae051b997ce7\": rpc error: code = NotFound desc = could not find container \"867fc8e3aacab21720f28a83a60d87a1999b0b1cb75cf4a6e7c6ae051b997ce7\": container with ID starting with 867fc8e3aacab21720f28a83a60d87a1999b0b1cb75cf4a6e7c6ae051b997ce7 not found: ID does not exist"
Nov 02 02:01:03.920775 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.920733    5875 scope.go:122] "RemoveContainer" containerID="5bbac46e87a02af71b7c6653d29432628bcdf0f3f0ae71c70b984912a2bd30c5"
Nov 02 02:01:03.932184 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.932164    5875 scope.go:122] "RemoveContainer" containerID="5bbac46e87a02af71b7c6653d29432628bcdf0f3f0ae71c70b984912a2bd30c5"
Nov 02 02:01:03.932549 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:01:03.932526    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bbac46e87a02af71b7c6653d29432628bcdf0f3f0ae71c70b984912a2bd30c5\": container with ID starting with 5bbac46e87a02af71b7c6653d29432628bcdf0f3f0ae71c70b984912a2bd30c5 not found: ID does not exist" containerID="5bbac46e87a02af71b7c6653d29432628bcdf0f3f0ae71c70b984912a2bd30c5"
Nov 02 02:01:03.932652 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.932557    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bbac46e87a02af71b7c6653d29432628bcdf0f3f0ae71c70b984912a2bd30c5"} err="failed to get container status \"5bbac46e87a02af71b7c6653d29432628bcdf0f3f0ae71c70b984912a2bd30c5\": rpc error: code = NotFound desc = could not find container \"5bbac46e87a02af71b7c6653d29432628bcdf0f3f0ae71c70b984912a2bd30c5\": container with ID starting with 5bbac46e87a02af71b7c6653d29432628bcdf0f3f0ae71c70b984912a2bd30c5 not found: ID does not exist"
Nov 02 02:01:03.932652 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.932580    5875 scope.go:122] "RemoveContainer" containerID="ca1bf69d08ce3a5c2eb3ec7b5bfd9869e3244d66b88d50cd5da80b992579494a"
Nov 02 02:01:03.950104 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.949997    5875 scope.go:122] "RemoveContainer" containerID="ca1bf69d08ce3a5c2eb3ec7b5bfd9869e3244d66b88d50cd5da80b992579494a"
Nov 02 02:01:03.950642 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:01:03.950617    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca1bf69d08ce3a5c2eb3ec7b5bfd9869e3244d66b88d50cd5da80b992579494a\": container with ID starting with ca1bf69d08ce3a5c2eb3ec7b5bfd9869e3244d66b88d50cd5da80b992579494a not found: ID does not exist" containerID="ca1bf69d08ce3a5c2eb3ec7b5bfd9869e3244d66b88d50cd5da80b992579494a"
Nov 02 02:01:03.950743 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.950654    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca1bf69d08ce3a5c2eb3ec7b5bfd9869e3244d66b88d50cd5da80b992579494a"} err="failed to get container status \"ca1bf69d08ce3a5c2eb3ec7b5bfd9869e3244d66b88d50cd5da80b992579494a\": rpc error: code = NotFound desc = could not find container \"ca1bf69d08ce3a5c2eb3ec7b5bfd9869e3244d66b88d50cd5da80b992579494a\": container with ID starting with ca1bf69d08ce3a5c2eb3ec7b5bfd9869e3244d66b88d50cd5da80b992579494a not found: ID does not exist"
Nov 02 02:01:03.950743 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.950678    5875 scope.go:122] "RemoveContainer" containerID="1408e3ce8a3aea7a7c66569514dc05028c2af17e9ffcce6b620da594484ce89a"
Nov 02 02:01:03.968906 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.968881    5875 scope.go:122] "RemoveContainer" containerID="1408e3ce8a3aea7a7c66569514dc05028c2af17e9ffcce6b620da594484ce89a"
Nov 02 02:01:03.969361 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:01:03.969331    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1408e3ce8a3aea7a7c66569514dc05028c2af17e9ffcce6b620da594484ce89a\": container with ID starting with 1408e3ce8a3aea7a7c66569514dc05028c2af17e9ffcce6b620da594484ce89a not found: ID does not exist" containerID="1408e3ce8a3aea7a7c66569514dc05028c2af17e9ffcce6b620da594484ce89a"
Nov 02 02:01:03.969473 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.969375    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1408e3ce8a3aea7a7c66569514dc05028c2af17e9ffcce6b620da594484ce89a"} err="failed to get container status \"1408e3ce8a3aea7a7c66569514dc05028c2af17e9ffcce6b620da594484ce89a\": rpc error: code = NotFound desc = could not find container \"1408e3ce8a3aea7a7c66569514dc05028c2af17e9ffcce6b620da594484ce89a\": container with ID starting with 1408e3ce8a3aea7a7c66569514dc05028c2af17e9ffcce6b620da594484ce89a not found: ID does not exist"
Nov 02 02:01:03.982490 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.982464    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-token\" (UniqueName: \"kubernetes.io/projected/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-token\") pod \"bceaf892-6e4e-4472-bf8d-34b1fe161036\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") "
Nov 02 02:01:03.982628 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.982519    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c637ee2b-9c6b-489e-8f6b-49112217a03c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c637ee2b-9c6b-489e-8f6b-49112217a03c-istiod-ca-cert\") pod \"c637ee2b-9c6b-489e-8f6b-49112217a03c\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") "
Nov 02 02:01:03.982628 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.982607    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c637ee2b-9c6b-489e-8f6b-49112217a03c-kube-api-access-44dh4\" (UniqueName: \"kubernetes.io/projected/c637ee2b-9c6b-489e-8f6b-49112217a03c-kube-api-access-44dh4\") pod \"c637ee2b-9c6b-489e-8f6b-49112217a03c\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") "
Nov 02 02:01:03.982761 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.982642    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-podinfo\") pod \"bceaf892-6e4e-4472-bf8d-34b1fe161036\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") "
Nov 02 02:01:03.982761 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.982676    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-workload-certs\") pod \"bceaf892-6e4e-4472-bf8d-34b1fe161036\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") "
Nov 02 02:01:03.982761 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.982717    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bceaf892-6e4e-4472-bf8d-34b1fe161036-kube-api-access-pzxmz\" (UniqueName: \"kubernetes.io/projected/bceaf892-6e4e-4472-bf8d-34b1fe161036-kube-api-access-pzxmz\") pod \"bceaf892-6e4e-4472-bf8d-34b1fe161036\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") "
Nov 02 02:01:03.982970 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.982824    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-credential-socket\") pod \"bceaf892-6e4e-4472-bf8d-34b1fe161036\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") "
Nov 02 02:01:03.982970 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.982861    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-workload-socket\") pod \"c637ee2b-9c6b-489e-8f6b-49112217a03c\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") "
Nov 02 02:01:03.982970 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.982912    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-workload-certs\") pod \"c637ee2b-9c6b-489e-8f6b-49112217a03c\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") "
Nov 02 02:01:03.982970 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.982951    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-credential-socket\") pod \"c637ee2b-9c6b-489e-8f6b-49112217a03c\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") "
Nov 02 02:01:03.983186 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.983005    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c637ee2b-9c6b-489e-8f6b-49112217a03c-istiod-ca-cert" pod "c637ee2b-9c6b-489e-8f6b-49112217a03c" (UID: "c637ee2b-9c6b-489e-8f6b-49112217a03c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:03.983186 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.983015    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-podinfo\") pod \"c637ee2b-9c6b-489e-8f6b-49112217a03c\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") "
Nov 02 02:01:03.983186 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.983063    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-envoy\") pod \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") "
Nov 02 02:01:03.983186 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.983126    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-credential-socket\") pod \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") "
Nov 02 02:01:03.983186 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.983159    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-data\") pod \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") "
Nov 02 02:01:03.983505 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.983213    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-kube-api-access-nm48c\" (UniqueName: \"kubernetes.io/projected/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-kube-api-access-nm48c\") pod \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") "
Nov 02 02:01:03.983647 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.983623    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-workload-certs" pod "c637ee2b-9c6b-489e-8f6b-49112217a03c" (UID: "c637ee2b-9c6b-489e-8f6b-49112217a03c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:03.983743 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.983653    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-credential-socket" pod "c637ee2b-9c6b-489e-8f6b-49112217a03c" (UID: "c637ee2b-9c6b-489e-8f6b-49112217a03c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.983858    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-credential-socket" pod "5ec9fbc9-a60c-4a82-932c-0af50b66ecab" (UID: "5ec9fbc9-a60c-4a82-932c-0af50b66ecab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.983909    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-credential-socket" pod "bceaf892-6e4e-4472-bf8d-34b1fe161036" (UID: "bceaf892-6e4e-4472-bf8d-34b1fe161036"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.983944    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-workload-certs" pod "bceaf892-6e4e-4472-bf8d-34b1fe161036" (UID: "bceaf892-6e4e-4472-bf8d-34b1fe161036"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.983975    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-workload-socket" pod "c637ee2b-9c6b-489e-8f6b-49112217a03c" (UID: "c637ee2b-9c6b-489e-8f6b-49112217a03c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.984209    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-data" pod "5ec9fbc9-a60c-4a82-932c-0af50b66ecab" (UID: "5ec9fbc9-a60c-4a82-932c-0af50b66ecab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.984285    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-token\" (UniqueName: \"kubernetes.io/projected/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-token\") pod \"c637ee2b-9c6b-489e-8f6b-49112217a03c\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") "
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.984336    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-envoy\") pod \"bceaf892-6e4e-4472-bf8d-34b1fe161036\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") "
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.984370    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-workload-certs\") pod \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") "
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.984403    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-data\") pod \"c637ee2b-9c6b-489e-8f6b-49112217a03c\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") "
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.984430    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-envoy\") pod \"c637ee2b-9c6b-489e-8f6b-49112217a03c\" (UID: \"c637ee2b-9c6b-489e-8f6b-49112217a03c\") "
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.984521    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bceaf892-6e4e-4472-bf8d-34b1fe161036-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bceaf892-6e4e-4472-bf8d-34b1fe161036-istiod-ca-cert\") pod \"bceaf892-6e4e-4472-bf8d-34b1fe161036\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") "
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.984558    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-podinfo\") pod \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") "
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.984586    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istiod-ca-cert\") pod \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") "
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.984632    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-workload-socket\") pod \"bceaf892-6e4e-4472-bf8d-34b1fe161036\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") "
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.984660    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-token\" (UniqueName: \"kubernetes.io/projected/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-token\") pod \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") "
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.984700    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-data\") pod \"bceaf892-6e4e-4472-bf8d-34b1fe161036\" (UID: \"bceaf892-6e4e-4472-bf8d-34b1fe161036\") "
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.984776    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-workload-socket\") pod \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\" (UID: \"5ec9fbc9-a60c-4a82-932c-0af50b66ecab\") "
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.985805    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c637ee2b-9c6b-489e-8f6b-49112217a03c-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.985830    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.985846    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.985860    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.985874    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.985887    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.985903    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.985918    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.986698    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-data" pod "c637ee2b-9c6b-489e-8f6b-49112217a03c" (UID: "c637ee2b-9c6b-489e-8f6b-49112217a03c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.986723    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-workload-socket" pod "5ec9fbc9-a60c-4a82-932c-0af50b66ecab" (UID: "5ec9fbc9-a60c-4a82-932c-0af50b66ecab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.986896    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-workload-socket" pod "bceaf892-6e4e-4472-bf8d-34b1fe161036" (UID: "bceaf892-6e4e-4472-bf8d-34b1fe161036"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.986928    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-podinfo" pod "c637ee2b-9c6b-489e-8f6b-49112217a03c" (UID: "c637ee2b-9c6b-489e-8f6b-49112217a03c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:01:03.986949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.986962    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-workload-certs" pod "5ec9fbc9-a60c-4a82-932c-0af50b66ecab" (UID: "5ec9fbc9-a60c-4a82-932c-0af50b66ecab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:03.988205 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.987252    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bceaf892-6e4e-4472-bf8d-34b1fe161036-kube-api-access-pzxmz" pod "bceaf892-6e4e-4472-bf8d-34b1fe161036" (UID: "bceaf892-6e4e-4472-bf8d-34b1fe161036"). InnerVolumeSpecName "kube-api-access-pzxmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:03.988205 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.987338    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bceaf892-6e4e-4472-bf8d-34b1fe161036-istiod-ca-cert" pod "bceaf892-6e4e-4472-bf8d-34b1fe161036" (UID: "bceaf892-6e4e-4472-bf8d-34b1fe161036"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:03.988205 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.987558    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-kube-api-access-nm48c" pod "5ec9fbc9-a60c-4a82-932c-0af50b66ecab" (UID: "5ec9fbc9-a60c-4a82-932c-0af50b66ecab"). InnerVolumeSpecName "kube-api-access-nm48c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:03.988205 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.987952    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istiod-ca-cert" pod "5ec9fbc9-a60c-4a82-932c-0af50b66ecab" (UID: "5ec9fbc9-a60c-4a82-932c-0af50b66ecab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:03.988205 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.988038    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-podinfo" pod "bceaf892-6e4e-4472-bf8d-34b1fe161036" (UID: "bceaf892-6e4e-4472-bf8d-34b1fe161036"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:01:03.988205 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.988100    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-token" pod "bceaf892-6e4e-4472-bf8d-34b1fe161036" (UID: "bceaf892-6e4e-4472-bf8d-34b1fe161036"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:03.988548 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.988515    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-data" pod "bceaf892-6e4e-4472-bf8d-34b1fe161036" (UID: "bceaf892-6e4e-4472-bf8d-34b1fe161036"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:03.988909 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.988890    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-token" pod "c637ee2b-9c6b-489e-8f6b-49112217a03c" (UID: "c637ee2b-9c6b-489e-8f6b-49112217a03c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:03.988986 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.988964    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-token" pod "5ec9fbc9-a60c-4a82-932c-0af50b66ecab" (UID: "5ec9fbc9-a60c-4a82-932c-0af50b66ecab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:03.990703 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.990675    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-envoy" pod "5ec9fbc9-a60c-4a82-932c-0af50b66ecab" (UID: "5ec9fbc9-a60c-4a82-932c-0af50b66ecab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:03.990792 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.990711    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-podinfo" pod "5ec9fbc9-a60c-4a82-932c-0af50b66ecab" (UID: "5ec9fbc9-a60c-4a82-932c-0af50b66ecab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:01:03.990792 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.990730    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c637ee2b-9c6b-489e-8f6b-49112217a03c-kube-api-access-44dh4" pod "c637ee2b-9c6b-489e-8f6b-49112217a03c" (UID: "c637ee2b-9c6b-489e-8f6b-49112217a03c"). InnerVolumeSpecName "kube-api-access-44dh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:03.990908 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.990886    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-envoy" pod "bceaf892-6e4e-4472-bf8d-34b1fe161036" (UID: "bceaf892-6e4e-4472-bf8d-34b1fe161036"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:03.992424 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:03.992399    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-envoy" pod "c637ee2b-9c6b-489e-8f6b-49112217a03c" (UID: "c637ee2b-9c6b-489e-8f6b-49112217a03c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:04.087537 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.087467    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.087537 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.087493    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.087537 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.087504    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm48c\" (UniqueName: \"kubernetes.io/projected/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-kube-api-access-nm48c\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.087537 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.087513    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.087537 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.087522    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.087537 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.087530    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.087537 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.087538    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.087865 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.087547    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c637ee2b-9c6b-489e-8f6b-49112217a03c-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.087865 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.087556    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bceaf892-6e4e-4472-bf8d-34b1fe161036-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.087865 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.087565    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.087865 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.087573    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.087865 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.087581    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.087865 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.087589    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.087865 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.087597    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.087865 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.087606    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec9fbc9-a60c-4a82-932c-0af50b66ecab-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.087865 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.087614    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.087865 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.087622    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44dh4\" (UniqueName: \"kubernetes.io/projected/c637ee2b-9c6b-489e-8f6b-49112217a03c-kube-api-access-44dh4\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.087865 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.087643    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bceaf892-6e4e-4472-bf8d-34b1fe161036-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.087865 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.087652    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzxmz\" (UniqueName: \"kubernetes.io/projected/bceaf892-6e4e-4472-bf8d-34b1fe161036-kube-api-access-pzxmz\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:04.240128 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.240092    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"]
Nov 02 02:01:04.245018 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.244990    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-v6g7b"]
Nov 02 02:01:04.678591 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.662097    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"]
Nov 02 02:01:04.757501 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.757471    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05f7e0b6-dc17-4891-9817-242be8465c54" path="/var/lib/kubelet/pods/05f7e0b6-dc17-4891-9817-242be8465c54/volumes"
Nov 02 02:01:04.758006 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.757992    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3726ebef-c581-4736-b0a3-ae417a6e8414" path="/var/lib/kubelet/pods/3726ebef-c581-4736-b0a3-ae417a6e8414/volumes"
Nov 02 02:01:04.758510 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.758494    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f4aa706-493e-42ec-a124-187020d8aa3e" path="/var/lib/kubelet/pods/3f4aa706-493e-42ec-a124-187020d8aa3e/volumes"
Nov 02 02:01:04.759295 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.759278    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b0c4e37-e032-4b48-8261-81294adf8635" path="/var/lib/kubelet/pods/6b0c4e37-e032-4b48-8261-81294adf8635/volumes"
Nov 02 02:01:04.759761 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.759748    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bef4b3e-a32d-4171-bd35-d19e9feef1e4" path="/var/lib/kubelet/pods/7bef4b3e-a32d-4171-bd35-d19e9feef1e4/volumes"
Nov 02 02:01:04.760065 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.760051    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4bbrw"]
Nov 02 02:01:04.812678 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.812653    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"]
Nov 02 02:01:04.816190 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.816170    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-4p9jt"]
Nov 02 02:01:04.899853 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.899820    5875 generic.go:356] "Generic (PLEG): container finished" podID="6181a5e7-b41f-4a80-a256-a2b609c0f551" containerID="f2d9690b23888988760664c3ad7ac8b0d867a1229a0502d44de05069bb6ff44c" exitCode=0
Nov 02 02:01:04.900017 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.899909    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc" event={"ID":"6181a5e7-b41f-4a80-a256-a2b609c0f551","Type":"ContainerDied","Data":"f2d9690b23888988760664c3ad7ac8b0d867a1229a0502d44de05069bb6ff44c"}
Nov 02 02:01:04.926623 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.926590    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"]
Nov 02 02:01:04.930613 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:04.930556    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-fbx4k"]
Nov 02 02:01:05.072345 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:05.072319    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc"
Nov 02 02:01:05.219298 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:05.219268    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6181a5e7-b41f-4a80-a256-a2b609c0f551-kube-api-access-fgsxl\" (UniqueName: \"kubernetes.io/projected/6181a5e7-b41f-4a80-a256-a2b609c0f551-kube-api-access-fgsxl\") pod \"6181a5e7-b41f-4a80-a256-a2b609c0f551\" (UID: \"6181a5e7-b41f-4a80-a256-a2b609c0f551\") "
Nov 02 02:01:05.219501 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:05.219408    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6181a5e7-b41f-4a80-a256-a2b609c0f551-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6181a5e7-b41f-4a80-a256-a2b609c0f551-maas-api-openshift-ca\") pod \"6181a5e7-b41f-4a80-a256-a2b609c0f551\" (UID: \"6181a5e7-b41f-4a80-a256-a2b609c0f551\") "
Nov 02 02:01:05.219501 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:05.219457    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6181a5e7-b41f-4a80-a256-a2b609c0f551-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6181a5e7-b41f-4a80-a256-a2b609c0f551-maas-api-xks-ca\") pod \"6181a5e7-b41f-4a80-a256-a2b609c0f551\" (UID: \"6181a5e7-b41f-4a80-a256-a2b609c0f551\") "
Nov 02 02:01:05.219850 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:05.219819    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6181a5e7-b41f-4a80-a256-a2b609c0f551-maas-api-openshift-ca" pod "6181a5e7-b41f-4a80-a256-a2b609c0f551" (UID: "6181a5e7-b41f-4a80-a256-a2b609c0f551"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:05.219972 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:05.219938    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6181a5e7-b41f-4a80-a256-a2b609c0f551-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:05.221803 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:05.221774    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6181a5e7-b41f-4a80-a256-a2b609c0f551-maas-api-xks-ca" pod "6181a5e7-b41f-4a80-a256-a2b609c0f551" (UID: "6181a5e7-b41f-4a80-a256-a2b609c0f551"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:01:05.222146 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:05.222125    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6181a5e7-b41f-4a80-a256-a2b609c0f551-kube-api-access-fgsxl" pod "6181a5e7-b41f-4a80-a256-a2b609c0f551" (UID: "6181a5e7-b41f-4a80-a256-a2b609c0f551"). InnerVolumeSpecName "kube-api-access-fgsxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:05.322076 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:05.322043    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6181a5e7-b41f-4a80-a256-a2b609c0f551-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:05.322076 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:05.322078    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgsxl\" (UniqueName: \"kubernetes.io/projected/6181a5e7-b41f-4a80-a256-a2b609c0f551-kube-api-access-fgsxl\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:05.912987 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:05.912952    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc" event={"ID":"6181a5e7-b41f-4a80-a256-a2b609c0f551","Type":"ContainerDied","Data":"52758232c521b739aa37142e78f9b2118b25eacf34ed355f9011f533e47445fd"}
Nov 02 02:01:05.913464 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:05.913008    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52758232c521b739aa37142e78f9b2118b25eacf34ed355f9011f533e47445fd"
Nov 02 02:01:05.913464 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:05.912960    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc"
Nov 02 02:01:06.485864 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:06.485838    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj"
Nov 02 02:01:06.486617 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:06.486598    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7"
Nov 02 02:01:06.697975 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:06.697939    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ec9fbc9-a60c-4a82-932c-0af50b66ecab" path="/var/lib/kubelet/pods/5ec9fbc9-a60c-4a82-932c-0af50b66ecab/volumes"
Nov 02 02:01:06.698557 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:06.698541    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bceaf892-6e4e-4472-bf8d-34b1fe161036" path="/var/lib/kubelet/pods/bceaf892-6e4e-4472-bf8d-34b1fe161036/volumes"
Nov 02 02:01:06.699078 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:06.699061    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c637ee2b-9c6b-489e-8f6b-49112217a03c" path="/var/lib/kubelet/pods/c637ee2b-9c6b-489e-8f6b-49112217a03c/volumes"
Nov 02 02:01:07.866313 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:07.866284    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6"
Nov 02 02:01:09.513604 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:09.513579    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb"
Nov 02 02:01:09.514027 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:09.513664    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz"
Nov 02 02:01:11.322276 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:01:11.322163    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:01:13.541604 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:13.541570    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"]
Nov 02 02:01:13.542127 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:13.541842    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf" podUID="24b5aa29-7f6a-46fb-a73f-6de2d308a2b1" containerName="istio-proxy" containerID="cri-o://cf821bd51f4731fc3820f3840d4a7b34742fde60166cf8fb948c9574598237f6" gracePeriod=30
Nov 02 02:01:13.591095 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:13.591049    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"]
Nov 02 02:01:13.591317 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:13.591282    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2" podUID="1a897476-f6fe-494d-b272-2d2f3fa3b08d" containerName="istio-proxy" containerID="cri-o://de57036e618880da6adb99d173919eb6aa6149811883ac014cb6542700e6564e" gracePeriod=30
Nov 02 02:01:13.594568 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:13.594532    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"]
Nov 02 02:01:13.594775 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:13.594753    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k" podUID="066ee18e-0d0d-4dee-a06c-3c6b218e1a80" containerName="istio-proxy" containerID="cri-o://75bd4f3dcfaf2f1aa05f5919387416fbac0300660582789cc733bd909cae9811" gracePeriod=30
Nov 02 02:01:13.641165 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:13.641134    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"]
Nov 02 02:01:13.641424 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:13.641386    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x" podUID="81b4f431-6173-4de9-b65c-f804391c02bb" containerName="istio-proxy" containerID="cri-o://aa04ba1197dd4fe03bf7dee9829967a78ca88eec5848b5500895c355be0ae760" gracePeriod=30
Nov 02 02:01:14.676383 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.676347    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q"]
Nov 02 02:01:14.677331 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677305    5875 cpu_manager.go:414] "containerMap: removing container" podUID="05f7e0b6-dc17-4891-9817-242be8465c54" containerName="istio-proxy"
Nov 02 02:01:14.677331 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677332    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f7e0b6-dc17-4891-9817-242be8465c54" containerName="istio-proxy"
Nov 02 02:01:14.677527 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677347    5875 cpu_manager.go:414] "containerMap: removing container" podUID="7bef4b3e-a32d-4171-bd35-d19e9feef1e4" containerName="istio-proxy"
Nov 02 02:01:14.677527 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677354    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bef4b3e-a32d-4171-bd35-d19e9feef1e4" containerName="istio-proxy"
Nov 02 02:01:14.677527 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677387    5875 cpu_manager.go:414] "containerMap: removing container" podUID="5ec9fbc9-a60c-4a82-932c-0af50b66ecab" containerName="istio-proxy"
Nov 02 02:01:14.677527 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677394    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ec9fbc9-a60c-4a82-932c-0af50b66ecab" containerName="istio-proxy"
Nov 02 02:01:14.677527 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677416    5875 cpu_manager.go:414] "containerMap: removing container" podUID="3f4aa706-493e-42ec-a124-187020d8aa3e" containerName="istio-proxy"
Nov 02 02:01:14.677527 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677423    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f4aa706-493e-42ec-a124-187020d8aa3e" containerName="istio-proxy"
Nov 02 02:01:14.677527 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677440    5875 cpu_manager.go:414] "containerMap: removing container" podUID="05f7e0b6-dc17-4891-9817-242be8465c54" containerName="istio-proxy"
Nov 02 02:01:14.677527 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677446    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f7e0b6-dc17-4891-9817-242be8465c54" containerName="istio-proxy"
Nov 02 02:01:14.677527 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677471    5875 cpu_manager.go:414] "containerMap: removing container" podUID="6181a5e7-b41f-4a80-a256-a2b609c0f551" containerName="revoke-keys"
Nov 02 02:01:14.677527 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677478    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6181a5e7-b41f-4a80-a256-a2b609c0f551" containerName="revoke-keys"
Nov 02 02:01:14.677527 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677490    5875 cpu_manager.go:414] "containerMap: removing container" podUID="6b0c4e37-e032-4b48-8261-81294adf8635" containerName="istio-proxy"
Nov 02 02:01:14.677527 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677497    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b0c4e37-e032-4b48-8261-81294adf8635" containerName="istio-proxy"
Nov 02 02:01:14.677527 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677513    5875 cpu_manager.go:414] "containerMap: removing container" podUID="3726ebef-c581-4736-b0a3-ae417a6e8414" containerName="istio-proxy"
Nov 02 02:01:14.678157 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677533    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3726ebef-c581-4736-b0a3-ae417a6e8414" containerName="istio-proxy"
Nov 02 02:01:14.678157 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677548    5875 cpu_manager.go:414] "containerMap: removing container" podUID="bceaf892-6e4e-4472-bf8d-34b1fe161036" containerName="istio-proxy"
Nov 02 02:01:14.678157 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677556    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bceaf892-6e4e-4472-bf8d-34b1fe161036" containerName="istio-proxy"
Nov 02 02:01:14.678157 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677566    5875 cpu_manager.go:414] "containerMap: removing container" podUID="c637ee2b-9c6b-489e-8f6b-49112217a03c" containerName="istio-proxy"
Nov 02 02:01:14.678157 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677573    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c637ee2b-9c6b-489e-8f6b-49112217a03c" containerName="istio-proxy"
Nov 02 02:01:14.678157 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677869    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f7e0b6-dc17-4891-9817-242be8465c54" containerName="istio-proxy"
Nov 02 02:01:14.678157 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677886    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bef4b3e-a32d-4171-bd35-d19e9feef1e4" containerName="istio-proxy"
Nov 02 02:01:14.678157 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677907    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ec9fbc9-a60c-4a82-932c-0af50b66ecab" containerName="istio-proxy"
Nov 02 02:01:14.678157 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677919    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f4aa706-493e-42ec-a124-187020d8aa3e" containerName="istio-proxy"
Nov 02 02:01:14.678157 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677934    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f7e0b6-dc17-4891-9817-242be8465c54" containerName="istio-proxy"
Nov 02 02:01:14.678157 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677945    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="6181a5e7-b41f-4a80-a256-a2b609c0f551" containerName="revoke-keys"
Nov 02 02:01:14.678157 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677953    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b0c4e37-e032-4b48-8261-81294adf8635" containerName="istio-proxy"
Nov 02 02:01:14.678157 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677963    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="3726ebef-c581-4736-b0a3-ae417a6e8414" containerName="istio-proxy"
Nov 02 02:01:14.678157 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677972    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="bceaf892-6e4e-4472-bf8d-34b1fe161036" containerName="istio-proxy"
Nov 02 02:01:14.678157 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.677981    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="c637ee2b-9c6b-489e-8f6b-49112217a03c" containerName="istio-proxy"
Nov 02 02:01:14.695210 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.693810    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q"
Nov 02 02:01:14.717774 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.717747    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d6edfb-c755-4afd-9237-1d4ce70352f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q\" (UID: \"43d6edfb-c755-4afd-9237-1d4ce70352f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q"
Nov 02 02:01:14.717896 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.717789    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9595v\" (UniqueName: \"kubernetes.io/projected/43d6edfb-c755-4afd-9237-1d4ce70352f1-kube-api-access-9595v\") pod \"payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q\" (UID: \"43d6edfb-c755-4afd-9237-1d4ce70352f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q"
Nov 02 02:01:14.726113 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.726083    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q"]
Nov 02 02:01:14.746616 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.746582    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-11f26c-dockercfg-cpjzb\""
Nov 02 02:01:14.757247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.747593    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-11f26c\""
Nov 02 02:01:14.917860 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.917814    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d6edfb-c755-4afd-9237-1d4ce70352f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q\" (UID: \"43d6edfb-c755-4afd-9237-1d4ce70352f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q"
Nov 02 02:01:14.941697 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.941613    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9595v\" (UniqueName: \"kubernetes.io/projected/43d6edfb-c755-4afd-9237-1d4ce70352f1-kube-api-access-9595v\") pod \"payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q\" (UID: \"43d6edfb-c755-4afd-9237-1d4ce70352f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q"
Nov 02 02:01:14.942950 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.922936    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d6edfb-c755-4afd-9237-1d4ce70352f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q\" (UID: \"43d6edfb-c755-4afd-9237-1d4ce70352f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q"
Nov 02 02:01:14.985242 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:14.975780    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq"]
Nov 02 02:01:15.018931 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:15.018905    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq"
Nov 02 02:01:15.028026 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:15.027998    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq"]
Nov 02 02:01:15.042959 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:15.042283    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9595v\" (UniqueName: \"kubernetes.io/projected/43d6edfb-c755-4afd-9237-1d4ce70352f1-kube-api-access-9595v\") pod \"payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q\" (UID: \"43d6edfb-c755-4afd-9237-1d4ce70352f1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q"
Nov 02 02:01:15.150205 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:15.150161    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/409169ea-5827-4946-9836-638b0bd62614-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq\" (UID: \"409169ea-5827-4946-9836-638b0bd62614\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq"
Nov 02 02:01:15.150413 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:15.150316    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96x6h\" (UniqueName: \"kubernetes.io/projected/409169ea-5827-4946-9836-638b0bd62614-kube-api-access-96x6h\") pod \"payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq\" (UID: \"409169ea-5827-4946-9836-638b0bd62614\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq"
Nov 02 02:01:15.252292 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:15.252261    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/409169ea-5827-4946-9836-638b0bd62614-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq\" (UID: \"409169ea-5827-4946-9836-638b0bd62614\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq"
Nov 02 02:01:15.252472 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:15.252346    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96x6h\" (UniqueName: \"kubernetes.io/projected/409169ea-5827-4946-9836-638b0bd62614-kube-api-access-96x6h\") pod \"payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq\" (UID: \"409169ea-5827-4946-9836-638b0bd62614\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq"
Nov 02 02:01:15.252966 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:15.252946    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/409169ea-5827-4946-9836-638b0bd62614-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq\" (UID: \"409169ea-5827-4946-9836-638b0bd62614\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq"
Nov 02 02:01:15.313066 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:15.313040    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96x6h\" (UniqueName: \"kubernetes.io/projected/409169ea-5827-4946-9836-638b0bd62614-kube-api-access-96x6h\") pod \"payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq\" (UID: \"409169ea-5827-4946-9836-638b0bd62614\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq"
Nov 02 02:01:15.314174 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:15.314151    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q"
Nov 02 02:01:15.355416 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:15.355389    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq"
Nov 02 02:01:15.699141 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:15.699114    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q"]
Nov 02 02:01:15.700007 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:01:15.699986    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d6edfb_c755_4afd_9237_1d4ce70352f1.slice/crio-269fd860842e703c2597239122b0d21d65fc8f1376adaf2fe5b79dc09efa4a53 WatchSource:0}: Error finding container 269fd860842e703c2597239122b0d21d65fc8f1376adaf2fe5b79dc09efa4a53: Status 404 returned error can't find the container with id 269fd860842e703c2597239122b0d21d65fc8f1376adaf2fe5b79dc09efa4a53
Nov 02 02:01:15.800144 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:01:15.800118    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409169ea_5827_4946_9836_638b0bd62614.slice/crio-368f7c4339fd837fd24bc0d8eb59febf85d70ce15a0ca3e3c3c05c0e6e65887e WatchSource:0}: Error finding container 368f7c4339fd837fd24bc0d8eb59febf85d70ce15a0ca3e3c3c05c0e6e65887e: Status 404 returned error can't find the container with id 368f7c4339fd837fd24bc0d8eb59febf85d70ce15a0ca3e3c3c05c0e6e65887e
Nov 02 02:01:15.800340 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:15.800317    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq"]
Nov 02 02:01:16.016779 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:16.016747    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq" event={"ID":"409169ea-5827-4946-9836-638b0bd62614","Type":"ContainerStarted","Data":"b973915c53c650330e0b9874b549b911489ba431a25c9dfbcc54259f8f987309"}
Nov 02 02:01:16.016955 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:16.016797    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq" event={"ID":"409169ea-5827-4946-9836-638b0bd62614","Type":"ContainerStarted","Data":"368f7c4339fd837fd24bc0d8eb59febf85d70ce15a0ca3e3c3c05c0e6e65887e"}
Nov 02 02:01:16.017010 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:16.016962    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq"
Nov 02 02:01:16.018682 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:16.018659    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q" event={"ID":"43d6edfb-c755-4afd-9237-1d4ce70352f1","Type":"ContainerStarted","Data":"6383bcca0e7e62e5acab828e8ba43df2438eca88f6ee7b7aeeed7e54649c4b6c"}
Nov 02 02:01:16.018820 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:16.018692    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q" event={"ID":"43d6edfb-c755-4afd-9237-1d4ce70352f1","Type":"ContainerStarted","Data":"269fd860842e703c2597239122b0d21d65fc8f1376adaf2fe5b79dc09efa4a53"}
Nov 02 02:01:16.018820 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:16.018743    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q"
Nov 02 02:01:16.153496 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:16.153422    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq" podStartSLOduration=2.153400816 podStartE2EDuration="2.153400816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:01:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:01:16.15287284 +0000 UTC m=+1723.593909338" watchObservedRunningTime="2026-11-02 02:01:16.153400816 +0000 UTC m=+1723.594437314"
Nov 02 02:01:16.279303 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:16.273282    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q" podStartSLOduration=2.2732610429999998 podStartE2EDuration="2.273261043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:01:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:01:16.269264619 +0000 UTC m=+1723.710301116" watchObservedRunningTime="2026-11-02 02:01:16.273261043 +0000 UTC m=+1723.714297541"
Nov 02 02:01:16.679693 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:16.679658    5875 patch_prober.go:29] interesting pod/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:01:16.679846 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:16.679728    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k" podUID="066ee18e-0d0d-4dee-a06c-3c6b218e1a80" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:01:18.914408 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:18.914374    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 02:01:18.945110 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:18.945078    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 02:01:19.022950 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.022911    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 02:01:19.035091 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.035069    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 02:01:19.082375 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.082340    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-data\") pod \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") "
Nov 02 02:01:19.082548 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.082412    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-workload-certs\") pod \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") "
Nov 02 02:01:19.082548 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.082508    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-workload-socket\") pod \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") "
Nov 02 02:01:19.082548 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.082538    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-credential-socket\") pod \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") "
Nov 02 02:01:19.082756 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.082567    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a897476-f6fe-494d-b272-2d2f3fa3b08d-kube-api-access-vh2nc\" (UniqueName: \"kubernetes.io/projected/1a897476-f6fe-494d-b272-2d2f3fa3b08d-kube-api-access-vh2nc\") pod \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") "
Nov 02 02:01:19.082756 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.082595    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-data\") pod \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") "
Nov 02 02:01:19.082756 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.082674    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-envoy\") pod \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") "
Nov 02 02:01:19.083059 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.082590    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-data" pod "24b5aa29-7f6a-46fb-a73f-6de2d308a2b1" (UID: "24b5aa29-7f6a-46fb-a73f-6de2d308a2b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:19.083059 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.082738    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-workload-certs" pod "1a897476-f6fe-494d-b272-2d2f3fa3b08d" (UID: "1a897476-f6fe-494d-b272-2d2f3fa3b08d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:19.083059 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.082769    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-podinfo\") pod \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") "
Nov 02 02:01:19.083059 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.082820    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-credential-socket" pod "1a897476-f6fe-494d-b272-2d2f3fa3b08d" (UID: "1a897476-f6fe-494d-b272-2d2f3fa3b08d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:19.083059 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.082836    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-workload-socket" pod "1a897476-f6fe-494d-b272-2d2f3fa3b08d" (UID: "1a897476-f6fe-494d-b272-2d2f3fa3b08d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:19.083059 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.082876    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-workload-certs\") pod \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") "
Nov 02 02:01:19.083059 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.082935    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-envoy\") pod \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") "
Nov 02 02:01:19.083059 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.083018    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-workload-socket\") pod \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") "
Nov 02 02:01:19.083059 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.083069    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istiod-ca-cert\") pod \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") "
Nov 02 02:01:19.083890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.083216    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-workload-certs" pod "24b5aa29-7f6a-46fb-a73f-6de2d308a2b1" (UID: "24b5aa29-7f6a-46fb-a73f-6de2d308a2b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:19.083890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.083513    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-podinfo\") pod \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") "
Nov 02 02:01:19.083890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.083567    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istiod-ca-cert\") pod \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") "
Nov 02 02:01:19.083890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.083630    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-credential-socket\") pod \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") "
Nov 02 02:01:19.083890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.083758    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-token\" (UniqueName: \"kubernetes.io/projected/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-token\") pod \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\" (UID: \"1a897476-f6fe-494d-b272-2d2f3fa3b08d\") "
Nov 02 02:01:19.083890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.083805    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-kube-api-access-694hm\" (UniqueName: \"kubernetes.io/projected/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-kube-api-access-694hm\") pod \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") "
Nov 02 02:01:19.083890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.083843    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-data" pod "1a897476-f6fe-494d-b272-2d2f3fa3b08d" (UID: "1a897476-f6fe-494d-b272-2d2f3fa3b08d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:19.086555 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.085919    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-token\" (UniqueName: \"kubernetes.io/projected/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-token\") pod \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\" (UID: \"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1\") "
Nov 02 02:01:19.086646 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.086620    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istiod-ca-cert" pod "1a897476-f6fe-494d-b272-2d2f3fa3b08d" (UID: "1a897476-f6fe-494d-b272-2d2f3fa3b08d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:19.087188 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.087026    5875 generic.go:356] "Generic (PLEG): container finished" podID="1a897476-f6fe-494d-b272-2d2f3fa3b08d" containerID="de57036e618880da6adb99d173919eb6aa6149811883ac014cb6542700e6564e" exitCode=0
Nov 02 02:01:19.087341 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.087202    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2" event={"ID":"1a897476-f6fe-494d-b272-2d2f3fa3b08d","Type":"ContainerDied","Data":"de57036e618880da6adb99d173919eb6aa6149811883ac014cb6542700e6564e"}
Nov 02 02:01:19.087341 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.087272    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2" event={"ID":"1a897476-f6fe-494d-b272-2d2f3fa3b08d","Type":"ContainerDied","Data":"edd62d3f57de5ff1a3d49db0c9dbc9e511882d32ada0ea9bde50a3dbe9919059"}
Nov 02 02:01:19.087341 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.087304    5875 scope.go:122] "RemoveContainer" containerID="de57036e618880da6adb99d173919eb6aa6149811883ac014cb6542700e6564e"
Nov 02 02:01:19.087534 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.087454    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"
Nov 02 02:01:19.087882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.087635    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.087882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.087668    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.087882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.087685    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.087882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.087700    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.087882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.087716    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.087882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.087732    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.087882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.087745    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.088733 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.088106    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-workload-socket" pod "24b5aa29-7f6a-46fb-a73f-6de2d308a2b1" (UID: "24b5aa29-7f6a-46fb-a73f-6de2d308a2b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:19.088733 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.088363    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istiod-ca-cert" pod "24b5aa29-7f6a-46fb-a73f-6de2d308a2b1" (UID: "24b5aa29-7f6a-46fb-a73f-6de2d308a2b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:19.088950 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.088910    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-credential-socket" pod "24b5aa29-7f6a-46fb-a73f-6de2d308a2b1" (UID: "24b5aa29-7f6a-46fb-a73f-6de2d308a2b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:19.092363 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.089875    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-podinfo" pod "1a897476-f6fe-494d-b272-2d2f3fa3b08d" (UID: "1a897476-f6fe-494d-b272-2d2f3fa3b08d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:01:19.092363 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.090600    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-token" pod "24b5aa29-7f6a-46fb-a73f-6de2d308a2b1" (UID: "24b5aa29-7f6a-46fb-a73f-6de2d308a2b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:19.092363 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.091360    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-envoy" pod "24b5aa29-7f6a-46fb-a73f-6de2d308a2b1" (UID: "24b5aa29-7f6a-46fb-a73f-6de2d308a2b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:19.093652 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.093629    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-kube-api-access-694hm" pod "24b5aa29-7f6a-46fb-a73f-6de2d308a2b1" (UID: "24b5aa29-7f6a-46fb-a73f-6de2d308a2b1"). InnerVolumeSpecName "kube-api-access-694hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:19.093803 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.093784    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-podinfo" pod "24b5aa29-7f6a-46fb-a73f-6de2d308a2b1" (UID: "24b5aa29-7f6a-46fb-a73f-6de2d308a2b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:01:19.093872 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.093792    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-token" pod "1a897476-f6fe-494d-b272-2d2f3fa3b08d" (UID: "1a897476-f6fe-494d-b272-2d2f3fa3b08d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:19.093872 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.093826    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a897476-f6fe-494d-b272-2d2f3fa3b08d-kube-api-access-vh2nc" pod "1a897476-f6fe-494d-b272-2d2f3fa3b08d" (UID: "1a897476-f6fe-494d-b272-2d2f3fa3b08d"). InnerVolumeSpecName "kube-api-access-vh2nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:19.095196 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.094949    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"
Nov 02 02:01:19.095196 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.094970    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x" event={"ID":"81b4f431-6173-4de9-b65c-f804391c02bb","Type":"ContainerDied","Data":"aa04ba1197dd4fe03bf7dee9829967a78ca88eec5848b5500895c355be0ae760"}
Nov 02 02:01:19.095196 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.094934    5875 generic.go:356] "Generic (PLEG): container finished" podID="81b4f431-6173-4de9-b65c-f804391c02bb" containerID="aa04ba1197dd4fe03bf7dee9829967a78ca88eec5848b5500895c355be0ae760" exitCode=0
Nov 02 02:01:19.095196 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.095160    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x" event={"ID":"81b4f431-6173-4de9-b65c-f804391c02bb","Type":"ContainerDied","Data":"bb3f401d2f18e28ef9b69772cb7e88a7fac6f58277fcdcf8feedba061e1a8453"}
Nov 02 02:01:19.096029 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.096004    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-envoy" pod "1a897476-f6fe-494d-b272-2d2f3fa3b08d" (UID: "1a897476-f6fe-494d-b272-2d2f3fa3b08d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:19.096805 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.096779    5875 generic.go:356] "Generic (PLEG): container finished" podID="24b5aa29-7f6a-46fb-a73f-6de2d308a2b1" containerID="cf821bd51f4731fc3820f3840d4a7b34742fde60166cf8fb948c9574598237f6" exitCode=0
Nov 02 02:01:19.096890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.096838    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"
Nov 02 02:01:19.096890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.096864    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf" event={"ID":"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1","Type":"ContainerDied","Data":"cf821bd51f4731fc3820f3840d4a7b34742fde60166cf8fb948c9574598237f6"}
Nov 02 02:01:19.097015 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.096920    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf" event={"ID":"24b5aa29-7f6a-46fb-a73f-6de2d308a2b1","Type":"ContainerDied","Data":"d42131f02b2350d12ee77e942604a636f739b3ca04b6291ec44c00adbe5b6b44"}
Nov 02 02:01:19.098426 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.098404    5875 generic.go:356] "Generic (PLEG): container finished" podID="066ee18e-0d0d-4dee-a06c-3c6b218e1a80" containerID="75bd4f3dcfaf2f1aa05f5919387416fbac0300660582789cc733bd909cae9811" exitCode=0
Nov 02 02:01:19.098541 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.098451    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"
Nov 02 02:01:19.098541 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.098463    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k" event={"ID":"066ee18e-0d0d-4dee-a06c-3c6b218e1a80","Type":"ContainerDied","Data":"75bd4f3dcfaf2f1aa05f5919387416fbac0300660582789cc733bd909cae9811"}
Nov 02 02:01:19.098541 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.098495    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k" event={"ID":"066ee18e-0d0d-4dee-a06c-3c6b218e1a80","Type":"ContainerDied","Data":"36db16eadc5869da0a0bbbb6c72a6121409c7c782ef08affe438a1e3723d0620"}
Nov 02 02:01:19.106060 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.106038    5875 scope.go:122] "RemoveContainer" containerID="de57036e618880da6adb99d173919eb6aa6149811883ac014cb6542700e6564e"
Nov 02 02:01:19.107094 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:01:19.106523    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de57036e618880da6adb99d173919eb6aa6149811883ac014cb6542700e6564e\": container with ID starting with de57036e618880da6adb99d173919eb6aa6149811883ac014cb6542700e6564e not found: ID does not exist" containerID="de57036e618880da6adb99d173919eb6aa6149811883ac014cb6542700e6564e"
Nov 02 02:01:19.107094 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.106557    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de57036e618880da6adb99d173919eb6aa6149811883ac014cb6542700e6564e"} err="failed to get container status \"de57036e618880da6adb99d173919eb6aa6149811883ac014cb6542700e6564e\": rpc error: code = NotFound desc = could not find container \"de57036e618880da6adb99d173919eb6aa6149811883ac014cb6542700e6564e\": container with ID starting with de57036e618880da6adb99d173919eb6aa6149811883ac014cb6542700e6564e not found: ID does not exist"
Nov 02 02:01:19.107094 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.106583    5875 scope.go:122] "RemoveContainer" containerID="aa04ba1197dd4fe03bf7dee9829967a78ca88eec5848b5500895c355be0ae760"
Nov 02 02:01:19.119049 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.118966    5875 scope.go:122] "RemoveContainer" containerID="aa04ba1197dd4fe03bf7dee9829967a78ca88eec5848b5500895c355be0ae760"
Nov 02 02:01:19.119450 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:01:19.119414    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa04ba1197dd4fe03bf7dee9829967a78ca88eec5848b5500895c355be0ae760\": container with ID starting with aa04ba1197dd4fe03bf7dee9829967a78ca88eec5848b5500895c355be0ae760 not found: ID does not exist" containerID="aa04ba1197dd4fe03bf7dee9829967a78ca88eec5848b5500895c355be0ae760"
Nov 02 02:01:19.119537 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.119448    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa04ba1197dd4fe03bf7dee9829967a78ca88eec5848b5500895c355be0ae760"} err="failed to get container status \"aa04ba1197dd4fe03bf7dee9829967a78ca88eec5848b5500895c355be0ae760\": rpc error: code = NotFound desc = could not find container \"aa04ba1197dd4fe03bf7dee9829967a78ca88eec5848b5500895c355be0ae760\": container with ID starting with aa04ba1197dd4fe03bf7dee9829967a78ca88eec5848b5500895c355be0ae760 not found: ID does not exist"
Nov 02 02:01:19.119537 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.119470    5875 scope.go:122] "RemoveContainer" containerID="cf821bd51f4731fc3820f3840d4a7b34742fde60166cf8fb948c9574598237f6"
Nov 02 02:01:19.129727 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.129705    5875 scope.go:122] "RemoveContainer" containerID="cf821bd51f4731fc3820f3840d4a7b34742fde60166cf8fb948c9574598237f6"
Nov 02 02:01:19.129998 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:01:19.129981    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf821bd51f4731fc3820f3840d4a7b34742fde60166cf8fb948c9574598237f6\": container with ID starting with cf821bd51f4731fc3820f3840d4a7b34742fde60166cf8fb948c9574598237f6 not found: ID does not exist" containerID="cf821bd51f4731fc3820f3840d4a7b34742fde60166cf8fb948c9574598237f6"
Nov 02 02:01:19.130078 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.130006    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf821bd51f4731fc3820f3840d4a7b34742fde60166cf8fb948c9574598237f6"} err="failed to get container status \"cf821bd51f4731fc3820f3840d4a7b34742fde60166cf8fb948c9574598237f6\": rpc error: code = NotFound desc = could not find container \"cf821bd51f4731fc3820f3840d4a7b34742fde60166cf8fb948c9574598237f6\": container with ID starting with cf821bd51f4731fc3820f3840d4a7b34742fde60166cf8fb948c9574598237f6 not found: ID does not exist"
Nov 02 02:01:19.130078 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.130028    5875 scope.go:122] "RemoveContainer" containerID="75bd4f3dcfaf2f1aa05f5919387416fbac0300660582789cc733bd909cae9811"
Nov 02 02:01:19.140460 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.140437    5875 scope.go:122] "RemoveContainer" containerID="75bd4f3dcfaf2f1aa05f5919387416fbac0300660582789cc733bd909cae9811"
Nov 02 02:01:19.140810 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:01:19.140790    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75bd4f3dcfaf2f1aa05f5919387416fbac0300660582789cc733bd909cae9811\": container with ID starting with 75bd4f3dcfaf2f1aa05f5919387416fbac0300660582789cc733bd909cae9811 not found: ID does not exist" containerID="75bd4f3dcfaf2f1aa05f5919387416fbac0300660582789cc733bd909cae9811"
Nov 02 02:01:19.140902 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.140821    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75bd4f3dcfaf2f1aa05f5919387416fbac0300660582789cc733bd909cae9811"} err="failed to get container status \"75bd4f3dcfaf2f1aa05f5919387416fbac0300660582789cc733bd909cae9811\": rpc error: code = NotFound desc = could not find container \"75bd4f3dcfaf2f1aa05f5919387416fbac0300660582789cc733bd909cae9811\": container with ID starting with 75bd4f3dcfaf2f1aa05f5919387416fbac0300660582789cc733bd909cae9811 not found: ID does not exist"
Nov 02 02:01:19.188515 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.188491    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-data\") pod \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") "
Nov 02 02:01:19.188632 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.188525    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-podinfo\") pod \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") "
Nov 02 02:01:19.188632 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.188599    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-workload-socket\") pod \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") "
Nov 02 02:01:19.188632 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.188630    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-credential-socket\") pod \"81b4f431-6173-4de9-b65c-f804391c02bb\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") "
Nov 02 02:01:19.188831 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.188661    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istiod-ca-cert\") pod \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") "
Nov 02 02:01:19.188831 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.188722    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-workload-certs\") pod \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") "
Nov 02 02:01:19.188831 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.188756    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81b4f431-6173-4de9-b65c-f804391c02bb-kube-api-access-cxclv\" (UniqueName: \"kubernetes.io/projected/81b4f431-6173-4de9-b65c-f804391c02bb-kube-api-access-cxclv\") pod \"81b4f431-6173-4de9-b65c-f804391c02bb\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") "
Nov 02 02:01:19.188952 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.188864    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81b4f431-6173-4de9-b65c-f804391c02bb-istio-token\" (UniqueName: \"kubernetes.io/projected/81b4f431-6173-4de9-b65c-f804391c02bb-istio-token\") pod \"81b4f431-6173-4de9-b65c-f804391c02bb\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") "
Nov 02 02:01:19.188952 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.188851    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-data" pod "066ee18e-0d0d-4dee-a06c-3c6b218e1a80" (UID: "066ee18e-0d0d-4dee-a06c-3c6b218e1a80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:19.188952 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.188892    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-credential-socket" pod "81b4f431-6173-4de9-b65c-f804391c02bb" (UID: "81b4f431-6173-4de9-b65c-f804391c02bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:19.188952 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.188895    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-workload-socket" pod "066ee18e-0d0d-4dee-a06c-3c6b218e1a80" (UID: "066ee18e-0d0d-4dee-a06c-3c6b218e1a80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:19.189097 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.188966    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-workload-socket\") pod \"81b4f431-6173-4de9-b65c-f804391c02bb\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") "
Nov 02 02:01:19.189097 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.189008    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-token\" (UniqueName: \"kubernetes.io/projected/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-token\") pod \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") "
Nov 02 02:01:19.189097 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.189072    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-istio-envoy\") pod \"81b4f431-6173-4de9-b65c-f804391c02bb\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") "
Nov 02 02:01:19.189266 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.189123    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-workload-certs" pod "066ee18e-0d0d-4dee-a06c-3c6b218e1a80" (UID: "066ee18e-0d0d-4dee-a06c-3c6b218e1a80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:19.189266 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.189130    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-envoy\") pod \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") "
Nov 02 02:01:19.189266 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.189145    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istiod-ca-cert" pod "066ee18e-0d0d-4dee-a06c-3c6b218e1a80" (UID: "066ee18e-0d0d-4dee-a06c-3c6b218e1a80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:19.189266 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.189184    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-istio-data\") pod \"81b4f431-6173-4de9-b65c-f804391c02bb\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") "
Nov 02 02:01:19.189471 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.189278    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-kube-api-access-frchm\" (UniqueName: \"kubernetes.io/projected/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-kube-api-access-frchm\") pod \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") "
Nov 02 02:01:19.189471 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.189284    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-workload-socket" pod "81b4f431-6173-4de9-b65c-f804391c02bb" (UID: "81b4f431-6173-4de9-b65c-f804391c02bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:19.189471 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.189310    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81b4f431-6173-4de9-b65c-f804391c02bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81b4f431-6173-4de9-b65c-f804391c02bb-istio-podinfo\") pod \"81b4f431-6173-4de9-b65c-f804391c02bb\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") "
Nov 02 02:01:19.189471 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.189338    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81b4f431-6173-4de9-b65c-f804391c02bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81b4f431-6173-4de9-b65c-f804391c02bb-istiod-ca-cert\") pod \"81b4f431-6173-4de9-b65c-f804391c02bb\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") "
Nov 02 02:01:19.189471 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.189413    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-workload-certs\") pod \"81b4f431-6173-4de9-b65c-f804391c02bb\" (UID: \"81b4f431-6173-4de9-b65c-f804391c02bb\") "
Nov 02 02:01:19.189471 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.189441    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-credential-socket\") pod \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\" (UID: \"066ee18e-0d0d-4dee-a06c-3c6b218e1a80\") "
Nov 02 02:01:19.189798 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.189487    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-istio-data" pod "81b4f431-6173-4de9-b65c-f804391c02bb" (UID: "81b4f431-6173-4de9-b65c-f804391c02bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:19.189798 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.189751    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b4f431-6173-4de9-b65c-f804391c02bb-istiod-ca-cert" pod "81b4f431-6173-4de9-b65c-f804391c02bb" (UID: "81b4f431-6173-4de9-b65c-f804391c02bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:19.189955 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.189930    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-credential-socket" pod "066ee18e-0d0d-4dee-a06c-3c6b218e1a80" (UID: "066ee18e-0d0d-4dee-a06c-3c6b218e1a80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:19.190149 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.190123    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-workload-certs" pod "81b4f431-6173-4de9-b65c-f804391c02bb" (UID: "81b4f431-6173-4de9-b65c-f804391c02bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:19.190378 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.190319    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81b4f431-6173-4de9-b65c-f804391c02bb-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.190378 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.190350    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.190378 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.190367    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-694hm\" (UniqueName: \"kubernetes.io/projected/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-kube-api-access-694hm\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.190378 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.190383    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.190378 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.190397    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.190378 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.190412    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.190378 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.190426    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.190378 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.190441    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.190378 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.190456    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.190378 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.190471    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.191210 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.190486    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh2nc\" (UniqueName: \"kubernetes.io/projected/1a897476-f6fe-494d-b272-2d2f3fa3b08d-kube-api-access-vh2nc\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.191210 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.190501    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.191210 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.190514    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.191210 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.190527    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a897476-f6fe-494d-b272-2d2f3fa3b08d-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.191210 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.190541    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.191210 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.190554    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.191210 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.190567    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.191210 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.190582    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.191210 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.190595    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.191210 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.190609    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.191210 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.190625    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.192072 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.191969    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-istio-envoy" pod "81b4f431-6173-4de9-b65c-f804391c02bb" (UID: "81b4f431-6173-4de9-b65c-f804391c02bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:19.192072 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.192039    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-envoy" pod "066ee18e-0d0d-4dee-a06c-3c6b218e1a80" (UID: "066ee18e-0d0d-4dee-a06c-3c6b218e1a80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:19.193755 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.192758    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b4f431-6173-4de9-b65c-f804391c02bb-istio-token" pod "81b4f431-6173-4de9-b65c-f804391c02bb" (UID: "81b4f431-6173-4de9-b65c-f804391c02bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:19.193755 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.193098    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b4f431-6173-4de9-b65c-f804391c02bb-kube-api-access-cxclv" pod "81b4f431-6173-4de9-b65c-f804391c02bb" (UID: "81b4f431-6173-4de9-b65c-f804391c02bb"). InnerVolumeSpecName "kube-api-access-cxclv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:19.194020 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.193983    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-podinfo" pod "066ee18e-0d0d-4dee-a06c-3c6b218e1a80" (UID: "066ee18e-0d0d-4dee-a06c-3c6b218e1a80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:01:19.194159 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.194098    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-kube-api-access-frchm" pod "066ee18e-0d0d-4dee-a06c-3c6b218e1a80" (UID: "066ee18e-0d0d-4dee-a06c-3c6b218e1a80"). InnerVolumeSpecName "kube-api-access-frchm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:19.195508 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.195447    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81b4f431-6173-4de9-b65c-f804391c02bb-istio-podinfo" pod "81b4f431-6173-4de9-b65c-f804391c02bb" (UID: "81b4f431-6173-4de9-b65c-f804391c02bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:01:19.200366 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.200314    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-token" pod "066ee18e-0d0d-4dee-a06c-3c6b218e1a80" (UID: "066ee18e-0d0d-4dee-a06c-3c6b218e1a80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:19.292489 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.292450    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81b4f431-6173-4de9-b65c-f804391c02bb-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.292489 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.292487    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.292725 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.292501    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81b4f431-6173-4de9-b65c-f804391c02bb-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.292725 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.292515    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.292725 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.292530    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frchm\" (UniqueName: \"kubernetes.io/projected/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-kube-api-access-frchm\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.292725 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.292547    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81b4f431-6173-4de9-b65c-f804391c02bb-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.292725 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.292561    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/066ee18e-0d0d-4dee-a06c-3c6b218e1a80-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.292725 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.292575    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxclv\" (UniqueName: \"kubernetes.io/projected/81b4f431-6173-4de9-b65c-f804391c02bb-kube-api-access-cxclv\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:19.628166 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.628070    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"]
Nov 02 02:01:19.636771 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.636730    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-9vg2x"]
Nov 02 02:01:19.741545 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.741510    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"]
Nov 02 02:01:19.744771 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.744746    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-jtr2k"]
Nov 02 02:01:19.853624 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.853588    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"]
Nov 02 02:01:19.858150 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.858125    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-rcns2"]
Nov 02 02:01:19.966424 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.966394    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"]
Nov 02 02:01:19.970885 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:19.970860    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c62671-openshift-default-7659bdbf95-98rrf"]
Nov 02 02:01:20.696372 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:20.696337    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="066ee18e-0d0d-4dee-a06c-3c6b218e1a80" path="/var/lib/kubelet/pods/066ee18e-0d0d-4dee-a06c-3c6b218e1a80/volumes"
Nov 02 02:01:20.696975 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:20.696959    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a897476-f6fe-494d-b272-2d2f3fa3b08d" path="/var/lib/kubelet/pods/1a897476-f6fe-494d-b272-2d2f3fa3b08d/volumes"
Nov 02 02:01:20.697554 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:20.697539    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24b5aa29-7f6a-46fb-a73f-6de2d308a2b1" path="/var/lib/kubelet/pods/24b5aa29-7f6a-46fb-a73f-6de2d308a2b1/volumes"
Nov 02 02:01:20.698570 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:20.698553    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81b4f431-6173-4de9-b65c-f804391c02bb" path="/var/lib/kubelet/pods/81b4f431-6173-4de9-b65c-f804391c02bb/volumes"
Nov 02 02:01:27.070866 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:27.070833    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq"
Nov 02 02:01:27.071281 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:27.071162    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q"
Nov 02 02:01:37.250586 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:37.250556    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:01:37.900825 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:37.900796    5875 scope.go:122] "RemoveContainer" containerID="3ebce0bf1c16d93e6c47fce1be93fe77531133a468d59dcdeb9ad746dc478a8e"
Nov 02 02:01:37.911638 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:37.911619    5875 scope.go:122] "RemoveContainer" containerID="504e451ff8c696d115cbccefe6cb8f7dd99fab94228540954cfb90f806b5ae74"
Nov 02 02:01:38.349089 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.349056    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr"]
Nov 02 02:01:38.349755 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.349740    5875 cpu_manager.go:414] "containerMap: removing container" podUID="81b4f431-6173-4de9-b65c-f804391c02bb" containerName="istio-proxy"
Nov 02 02:01:38.349800 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.349757    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b4f431-6173-4de9-b65c-f804391c02bb" containerName="istio-proxy"
Nov 02 02:01:38.349836 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.349808    5875 cpu_manager.go:414] "containerMap: removing container" podUID="1a897476-f6fe-494d-b272-2d2f3fa3b08d" containerName="istio-proxy"
Nov 02 02:01:38.349836 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.349813    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a897476-f6fe-494d-b272-2d2f3fa3b08d" containerName="istio-proxy"
Nov 02 02:01:38.349836 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.349823    5875 cpu_manager.go:414] "containerMap: removing container" podUID="066ee18e-0d0d-4dee-a06c-3c6b218e1a80" containerName="istio-proxy"
Nov 02 02:01:38.349836 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.349828    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="066ee18e-0d0d-4dee-a06c-3c6b218e1a80" containerName="istio-proxy"
Nov 02 02:01:38.349967 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.349843    5875 cpu_manager.go:414] "containerMap: removing container" podUID="24b5aa29-7f6a-46fb-a73f-6de2d308a2b1" containerName="istio-proxy"
Nov 02 02:01:38.349967 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.349848    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b5aa29-7f6a-46fb-a73f-6de2d308a2b1" containerName="istio-proxy"
Nov 02 02:01:38.349967 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.349931    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b5aa29-7f6a-46fb-a73f-6de2d308a2b1" containerName="istio-proxy"
Nov 02 02:01:38.349967 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.349939    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b4f431-6173-4de9-b65c-f804391c02bb" containerName="istio-proxy"
Nov 02 02:01:38.349967 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.349955    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a897476-f6fe-494d-b272-2d2f3fa3b08d" containerName="istio-proxy"
Nov 02 02:01:38.349967 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.349962    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="066ee18e-0d0d-4dee-a06c-3c6b218e1a80" containerName="istio-proxy"
Nov 02 02:01:38.356991 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.356966    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr"
Nov 02 02:01:38.407245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.404990    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/314d6835-e1c6-48ba-b4eb-a744a699c061-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr\" (UID: \"314d6835-e1c6-48ba-b4eb-a744a699c061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr"
Nov 02 02:01:38.407245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.405061    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/314d6835-e1c6-48ba-b4eb-a744a699c061-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr\" (UID: \"314d6835-e1c6-48ba-b4eb-a744a699c061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr"
Nov 02 02:01:38.407245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.405129    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhr6v\" (UniqueName: \"kubernetes.io/projected/314d6835-e1c6-48ba-b4eb-a744a699c061-kube-api-access-lhr6v\") pod \"maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr\" (UID: \"314d6835-e1c6-48ba-b4eb-a744a699c061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr"
Nov 02 02:01:38.407245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.405278    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/314d6835-e1c6-48ba-b4eb-a744a699c061-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr\" (UID: \"314d6835-e1c6-48ba-b4eb-a744a699c061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr"
Nov 02 02:01:38.417288 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.415199    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr"]
Nov 02 02:01:38.457904 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.457869    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-129de4-serving-cert\""
Nov 02 02:01:38.528250 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.525065    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhr6v\" (UniqueName: \"kubernetes.io/projected/314d6835-e1c6-48ba-b4eb-a744a699c061-kube-api-access-lhr6v\") pod \"maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr\" (UID: \"314d6835-e1c6-48ba-b4eb-a744a699c061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr"
Nov 02 02:01:38.528250 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.525205    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/314d6835-e1c6-48ba-b4eb-a744a699c061-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr\" (UID: \"314d6835-e1c6-48ba-b4eb-a744a699c061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr"
Nov 02 02:01:38.528250 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.525602    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/314d6835-e1c6-48ba-b4eb-a744a699c061-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr\" (UID: \"314d6835-e1c6-48ba-b4eb-a744a699c061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr"
Nov 02 02:01:38.528250 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.525654    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/314d6835-e1c6-48ba-b4eb-a744a699c061-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr\" (UID: \"314d6835-e1c6-48ba-b4eb-a744a699c061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr"
Nov 02 02:01:38.528250 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.526625    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/314d6835-e1c6-48ba-b4eb-a744a699c061-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr\" (UID: \"314d6835-e1c6-48ba-b4eb-a744a699c061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr"
Nov 02 02:01:38.528250 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:01:38.527046    5875 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-serving-cert: secret "maas-api-e2e-shared-a-w0-129de4-serving-cert" not found
Nov 02 02:01:38.528250 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:01:38.527112    5875 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/314d6835-e1c6-48ba-b4eb-a744a699c061-maas-api-tls podName:314d6835-e1c6-48ba-b4eb-a744a699c061 nodeName:}" failed. No retries permitted until 2026-11-02 02:01:39.027092522 +0000 UTC m=+1746.468129012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/314d6835-e1c6-48ba-b4eb-a744a699c061-maas-api-tls") pod "maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr" (UID: "314d6835-e1c6-48ba-b4eb-a744a699c061") : secret "maas-api-e2e-shared-a-w0-129de4-serving-cert" not found
Nov 02 02:01:38.530854 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.530825    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/314d6835-e1c6-48ba-b4eb-a744a699c061-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr\" (UID: \"314d6835-e1c6-48ba-b4eb-a744a699c061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr"
Nov 02 02:01:38.587262 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:38.587212    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhr6v\" (UniqueName: \"kubernetes.io/projected/314d6835-e1c6-48ba-b4eb-a744a699c061-kube-api-access-lhr6v\") pod \"maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr\" (UID: \"314d6835-e1c6-48ba-b4eb-a744a699c061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr"
Nov 02 02:01:39.048955 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:39.048917    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/314d6835-e1c6-48ba-b4eb-a744a699c061-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr\" (UID: \"314d6835-e1c6-48ba-b4eb-a744a699c061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr"
Nov 02 02:01:39.051693 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:39.051662    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/314d6835-e1c6-48ba-b4eb-a744a699c061-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr\" (UID: \"314d6835-e1c6-48ba-b4eb-a744a699c061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr"
Nov 02 02:01:39.282593 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:39.282556    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr"
Nov 02 02:01:39.695643 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:39.695612    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr"]
Nov 02 02:01:40.269114 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:40.269072    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr" event={"ID":"314d6835-e1c6-48ba-b4eb-a744a699c061","Type":"ContainerStarted","Data":"5075a892a26d5365ab61494d813f6a5d1c8880eb3c7273d65e0d9d4b3403ca88"}
Nov 02 02:01:41.277326 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:41.277290    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr" event={"ID":"314d6835-e1c6-48ba-b4eb-a744a699c061","Type":"ContainerStarted","Data":"383100b6d34c35b3ca81bea70223dada3904205ca988169e5ffe6604b6e1dc25"}
Nov 02 02:01:41.277730 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:41.277375    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr"
Nov 02 02:01:41.345047 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:01:41.345009    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:01:41.397014 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:41.396962    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr" podStartSLOduration=2.501689903 podStartE2EDuration="3.396947605s" totalImagesPullingTime="895.257702ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:01:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:01:41.39410758 +0000 UTC m=+1748.835144077" watchObservedRunningTime="2026-11-02 02:01:41.396947605 +0000 UTC m=+1748.837984101"
Nov 02 02:01:47.292237 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:47.292178    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr"
Nov 02 02:01:48.338537 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:48.338504    5875 generic.go:356] "Generic (PLEG): container finished" podID="c764b207-bd1e-4652-b4c1-519c8f453aa2" containerID="c06e8a7212c31d1e56abba5d488e65c1f67384f5d258f35e8f3dfa83c267712c" exitCode=0
Nov 02 02:01:48.339084 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:48.338587    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs" event={"ID":"c764b207-bd1e-4652-b4c1-519c8f453aa2","Type":"ContainerDied","Data":"c06e8a7212c31d1e56abba5d488e65c1f67384f5d258f35e8f3dfa83c267712c"}
Nov 02 02:01:48.339084 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:48.339031    5875 scope.go:122] "RemoveContainer" containerID="c06e8a7212c31d1e56abba5d488e65c1f67384f5d258f35e8f3dfa83c267712c"
Nov 02 02:01:48.340679 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:48.340646    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:01:48.341073 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:48.340997    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:01:48.341178 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:48.341118    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:01:49.349635 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:49.349591    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs" event={"ID":"c764b207-bd1e-4652-b4c1-519c8f453aa2","Type":"ContainerStarted","Data":"0b62cee84c26747b0a06eaec209471c1f8029cb967b1598169949f1ef40ea83f"}
Nov 02 02:01:49.372706 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:49.372674    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd"]
Nov 02 02:01:49.377575 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:49.377557    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd"
Nov 02 02:01:49.432316 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:49.432279    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-8e79d7-serving-cert\""
Nov 02 02:01:49.469670 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:49.469637    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5c9f74d-7015-401f-900b-d065938f4507-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd\" (UID: \"a5c9f74d-7015-401f-900b-d065938f4507\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd"
Nov 02 02:01:49.469857 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:49.469792    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5c9f74d-7015-401f-900b-d065938f4507-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd\" (UID: \"a5c9f74d-7015-401f-900b-d065938f4507\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd"
Nov 02 02:01:49.469904 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:49.469855    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5c9f74d-7015-401f-900b-d065938f4507-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd\" (UID: \"a5c9f74d-7015-401f-900b-d065938f4507\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd"
Nov 02 02:01:49.469997 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:49.469984    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7ghr\" (UniqueName: \"kubernetes.io/projected/a5c9f74d-7015-401f-900b-d065938f4507-kube-api-access-m7ghr\") pod \"maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd\" (UID: \"a5c9f74d-7015-401f-900b-d065938f4507\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd"
Nov 02 02:01:49.481559 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:49.481526    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd"]
Nov 02 02:01:49.571095 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:49.571053    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5c9f74d-7015-401f-900b-d065938f4507-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd\" (UID: \"a5c9f74d-7015-401f-900b-d065938f4507\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd"
Nov 02 02:01:49.571294 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:49.571141    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5c9f74d-7015-401f-900b-d065938f4507-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd\" (UID: \"a5c9f74d-7015-401f-900b-d065938f4507\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd"
Nov 02 02:01:49.571294 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:49.571279    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7ghr\" (UniqueName: \"kubernetes.io/projected/a5c9f74d-7015-401f-900b-d065938f4507-kube-api-access-m7ghr\") pod \"maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd\" (UID: \"a5c9f74d-7015-401f-900b-d065938f4507\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd"
Nov 02 02:01:49.571436 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:49.571370    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5c9f74d-7015-401f-900b-d065938f4507-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd\" (UID: \"a5c9f74d-7015-401f-900b-d065938f4507\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd"
Nov 02 02:01:49.571555 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:01:49.571525    5875 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-serving-cert: secret "maas-api-e2e-shared-b-w5-8e79d7-serving-cert" not found
Nov 02 02:01:49.571610 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:01:49.571597    5875 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5c9f74d-7015-401f-900b-d065938f4507-maas-api-tls podName:a5c9f74d-7015-401f-900b-d065938f4507 nodeName:}" failed. No retries permitted until 2026-11-02 02:01:50.071575622 +0000 UTC m=+1757.512612099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a5c9f74d-7015-401f-900b-d065938f4507-maas-api-tls") pod "maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd" (UID: "a5c9f74d-7015-401f-900b-d065938f4507") : secret "maas-api-e2e-shared-b-w5-8e79d7-serving-cert" not found
Nov 02 02:01:49.572147 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:49.572097    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5c9f74d-7015-401f-900b-d065938f4507-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd\" (UID: \"a5c9f74d-7015-401f-900b-d065938f4507\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd"
Nov 02 02:01:49.573962 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:49.573937    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5c9f74d-7015-401f-900b-d065938f4507-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd\" (UID: \"a5c9f74d-7015-401f-900b-d065938f4507\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd"
Nov 02 02:01:49.633647 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:49.633570    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7ghr\" (UniqueName: \"kubernetes.io/projected/a5c9f74d-7015-401f-900b-d065938f4507-kube-api-access-m7ghr\") pod \"maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd\" (UID: \"a5c9f74d-7015-401f-900b-d065938f4507\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd"
Nov 02 02:01:49.662962 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:49.662925    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 02:01:49.664156 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:49.664124    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 02 02:01:49.664326 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:49.664182    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs" podUID="c764b207-bd1e-4652-b4c1-519c8f453aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 02 02:01:50.028374 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:50.028317    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc"]
Nov 02 02:01:50.031452 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:50.031412    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25789ee1-fg9nc"]
Nov 02 02:01:50.091628 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:50.091600    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5c9f74d-7015-401f-900b-d065938f4507-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd\" (UID: \"a5c9f74d-7015-401f-900b-d065938f4507\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd"
Nov 02 02:01:50.094111 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:50.094088    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5c9f74d-7015-401f-900b-d065938f4507-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd\" (UID: \"a5c9f74d-7015-401f-900b-d065938f4507\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd"
Nov 02 02:01:50.291873 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:50.291792    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd"
Nov 02 02:01:50.663775 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:50.663692    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 02 02:01:50.663775 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:50.663749    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs" podUID="c764b207-bd1e-4652-b4c1-519c8f453aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 02 02:01:50.696488 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:50.696456    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6181a5e7-b41f-4a80-a256-a2b609c0f551" path="/var/lib/kubelet/pods/6181a5e7-b41f-4a80-a256-a2b609c0f551/volumes"
Nov 02 02:01:50.725180 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:50.725080    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd"]
Nov 02 02:01:50.731254 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:01:50.731195    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c9f74d_7015_401f_900b_d065938f4507.slice/crio-a3c1c35555491070a5def47d62614d42b1592885ad51ddb942588f8442b48318 WatchSource:0}: Error finding container a3c1c35555491070a5def47d62614d42b1592885ad51ddb942588f8442b48318: Status 404 returned error can't find the container with id a3c1c35555491070a5def47d62614d42b1592885ad51ddb942588f8442b48318
Nov 02 02:01:51.370325 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:51.370295    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd" event={"ID":"a5c9f74d-7015-401f-900b-d065938f4507","Type":"ContainerStarted","Data":"a3c1c35555491070a5def47d62614d42b1592885ad51ddb942588f8442b48318"}
Nov 02 02:01:51.663672 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:51.663519    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 02 02:01:51.663672 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:51.663572    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs" podUID="c764b207-bd1e-4652-b4c1-519c8f453aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 02 02:01:52.379357 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:52.379317    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd" event={"ID":"a5c9f74d-7015-401f-900b-d065938f4507","Type":"ContainerStarted","Data":"b896f0500cc649a6f494f98756c4e1f88726f2b8f64db991e95ea5998bde6093"}
Nov 02 02:01:52.379747 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:52.379438    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd"
Nov 02 02:01:52.526001 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:52.525920    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd" podStartSLOduration=2.696549982 podStartE2EDuration="3.525896927s" totalImagesPullingTime="829.346945ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:01:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:01:52.497184754 +0000 UTC m=+1759.938221253" watchObservedRunningTime="2026-11-02 02:01:52.525896927 +0000 UTC m=+1759.966933425"
Nov 02 02:01:52.666168 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:52.665652    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 02 02:01:52.666168 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:52.665714    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs" podUID="c764b207-bd1e-4652-b4c1-519c8f453aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 02 02:01:52.783771 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:52.783738    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 02:01:53.662719 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:53.662683    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 02 02:01:53.663109 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:53.662735    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs" podUID="c764b207-bd1e-4652-b4c1-519c8f453aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 02 02:01:54.663205 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:54.663170    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 02 02:01:54.663765 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:54.663271    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs" podUID="c764b207-bd1e-4652-b4c1-519c8f453aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 02 02:01:54.902079 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:54.902039    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz"]
Nov 02 02:01:54.906557 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:54.906535    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz"
Nov 02 02:01:55.005981 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:55.005953    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-c1aad5-serving-cert\""
Nov 02 02:01:55.009577 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:55.009551    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz"]
Nov 02 02:01:55.043587 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:55.043554    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99df19ae-03ba-4b87-ae0d-8d837d693d7d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz\" (UID: \"99df19ae-03ba-4b87-ae0d-8d837d693d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz"
Nov 02 02:01:55.043776 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:55.043616    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q6zv\" (UniqueName: \"kubernetes.io/projected/99df19ae-03ba-4b87-ae0d-8d837d693d7d-kube-api-access-6q6zv\") pod \"maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz\" (UID: \"99df19ae-03ba-4b87-ae0d-8d837d693d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz"
Nov 02 02:01:55.043878 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:55.043816    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99df19ae-03ba-4b87-ae0d-8d837d693d7d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz\" (UID: \"99df19ae-03ba-4b87-ae0d-8d837d693d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz"
Nov 02 02:01:55.043947 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:55.043917    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99df19ae-03ba-4b87-ae0d-8d837d693d7d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz\" (UID: \"99df19ae-03ba-4b87-ae0d-8d837d693d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz"
Nov 02 02:01:55.145158 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:55.145117    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99df19ae-03ba-4b87-ae0d-8d837d693d7d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz\" (UID: \"99df19ae-03ba-4b87-ae0d-8d837d693d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz"
Nov 02 02:01:55.145412 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:55.145170    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q6zv\" (UniqueName: \"kubernetes.io/projected/99df19ae-03ba-4b87-ae0d-8d837d693d7d-kube-api-access-6q6zv\") pod \"maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz\" (UID: \"99df19ae-03ba-4b87-ae0d-8d837d693d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz"
Nov 02 02:01:55.145412 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:55.145245    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99df19ae-03ba-4b87-ae0d-8d837d693d7d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz\" (UID: \"99df19ae-03ba-4b87-ae0d-8d837d693d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz"
Nov 02 02:01:55.145412 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:55.145342    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99df19ae-03ba-4b87-ae0d-8d837d693d7d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz\" (UID: \"99df19ae-03ba-4b87-ae0d-8d837d693d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz"
Nov 02 02:01:55.145570 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:01:55.145430    5875 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-serving-cert: secret "maas-api-e2e-shared-a-w2-c1aad5-serving-cert" not found
Nov 02 02:01:55.145570 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:01:55.145511    5875 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99df19ae-03ba-4b87-ae0d-8d837d693d7d-maas-api-tls podName:99df19ae-03ba-4b87-ae0d-8d837d693d7d nodeName:}" failed. No retries permitted until 2026-11-02 02:01:55.645489464 +0000 UTC m=+1763.086525957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/99df19ae-03ba-4b87-ae0d-8d837d693d7d-maas-api-tls") pod "maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz" (UID: "99df19ae-03ba-4b87-ae0d-8d837d693d7d") : secret "maas-api-e2e-shared-a-w2-c1aad5-serving-cert" not found
Nov 02 02:01:55.146010 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:55.145990    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99df19ae-03ba-4b87-ae0d-8d837d693d7d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz\" (UID: \"99df19ae-03ba-4b87-ae0d-8d837d693d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz"
Nov 02 02:01:55.147748 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:55.147727    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99df19ae-03ba-4b87-ae0d-8d837d693d7d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz\" (UID: \"99df19ae-03ba-4b87-ae0d-8d837d693d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz"
Nov 02 02:01:55.208138 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:55.208104    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q6zv\" (UniqueName: \"kubernetes.io/projected/99df19ae-03ba-4b87-ae0d-8d837d693d7d-kube-api-access-6q6zv\") pod \"maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz\" (UID: \"99df19ae-03ba-4b87-ae0d-8d837d693d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz"
Nov 02 02:01:55.653761 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:55.653671    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99df19ae-03ba-4b87-ae0d-8d837d693d7d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz\" (UID: \"99df19ae-03ba-4b87-ae0d-8d837d693d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz"
Nov 02 02:01:55.656333 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:55.656310    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99df19ae-03ba-4b87-ae0d-8d837d693d7d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz\" (UID: \"99df19ae-03ba-4b87-ae0d-8d837d693d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz"
Nov 02 02:01:55.668052 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:55.668030    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 02:01:55.669771 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:55.669752    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 02:01:55.820897 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:55.820867    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz"
Nov 02 02:01:56.207167 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:56.207136    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz"]
Nov 02 02:01:56.208413 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:01:56.208387    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99df19ae_03ba_4b87_ae0d_8d837d693d7d.slice/crio-3708071aa8f98b7b286618412e4ead368effc75e35b8c254ee94e2f8705d5ddf WatchSource:0}: Error finding container 3708071aa8f98b7b286618412e4ead368effc75e35b8c254ee94e2f8705d5ddf: Status 404 returned error can't find the container with id 3708071aa8f98b7b286618412e4ead368effc75e35b8c254ee94e2f8705d5ddf
Nov 02 02:01:56.432405 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:56.432373    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz" event={"ID":"99df19ae-03ba-4b87-ae0d-8d837d693d7d","Type":"ContainerStarted","Data":"3708071aa8f98b7b286618412e4ead368effc75e35b8c254ee94e2f8705d5ddf"}
Nov 02 02:01:56.972992 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:56.972956    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"]
Nov 02 02:01:56.973497 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:56.973209    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f" podUID="eedec700-8816-4b96-a087-7d0d691123bf" containerName="istio-proxy" containerID="cri-o://e2c1c23aa5ab53e5abc5f336a306437262145fecf70ce32c55303cdfc05a87ee" gracePeriod=30
Nov 02 02:01:57.081563 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:57.081528    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f30cd23a-b8fj2"]
Nov 02 02:01:57.094445 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:57.094416    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f30cd23a-b8fj2"]
Nov 02 02:01:57.470564 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:57.470523    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz" event={"ID":"99df19ae-03ba-4b87-ae0d-8d837d693d7d","Type":"ContainerStarted","Data":"1a7e3350eb4cef38949f56de73a434e350c8331003f99d5186f83042a4a072bb"}
Nov 02 02:01:57.472082 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:57.472054    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz"
Nov 02 02:01:57.592595 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:57.592490    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz" podStartSLOduration=2.6167467589999998 podStartE2EDuration="3.592476092s" totalImagesPullingTime="975.729333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:01:57.589197287 +0000 UTC m=+1765.030233861" watchObservedRunningTime="2026-11-02 02:01:57.592476092 +0000 UTC m=+1765.033512588"
Nov 02 02:01:58.392809 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:58.392780    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd"
Nov 02 02:01:58.699138 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:58.699050    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e898b37-cbe5-427e-9c6c-563dedfc662e" path="/var/lib/kubelet/pods/7e898b37-cbe5-427e-9c6c-563dedfc662e/volumes"
Nov 02 02:01:58.963739 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:58.963708    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"]
Nov 02 02:01:58.968503 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:58.968483    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.021113 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.021080    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-bd3971-openshift-default-dockercfg-j8pgn\""
Nov 02 02:01:59.051766 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.051732    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-istio-envoy\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.051942 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.051775    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-istio-data\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.052018 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.051929    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-workload-socket\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.052018 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.051994    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-workload-certs\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.052210 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.052186    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160fc571-588a-47fd-9acc-f82010429400-istio-podinfo\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.052331 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.052308    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-credential-socket\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.052404 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.052388    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160fc571-588a-47fd-9acc-f82010429400-istiod-ca-cert\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.052461 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.052434    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/160fc571-588a-47fd-9acc-f82010429400-istio-token\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.052519 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.052495    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nx4p\" (UniqueName: \"kubernetes.io/projected/160fc571-588a-47fd-9acc-f82010429400-kube-api-access-8nx4p\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.073447 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.073416    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"]
Nov 02 02:01:59.154610 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.154566    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160fc571-588a-47fd-9acc-f82010429400-istio-podinfo\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.154819 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.154797    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-credential-socket\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.155131 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.155084    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160fc571-588a-47fd-9acc-f82010429400-istiod-ca-cert\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.155397 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.155363    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/160fc571-588a-47fd-9acc-f82010429400-istio-token\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.155767 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.155748    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nx4p\" (UniqueName: \"kubernetes.io/projected/160fc571-588a-47fd-9acc-f82010429400-kube-api-access-8nx4p\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.156026 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.156003    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-istio-envoy\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.156193 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.156174    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-istio-data\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.156436 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.156413    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-workload-socket\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.156671 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.156649    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-workload-certs\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.157585 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.157478    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-credential-socket\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.158335 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.157892    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160fc571-588a-47fd-9acc-f82010429400-istiod-ca-cert\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.158474 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.158394    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-workload-socket\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.158705 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.158676    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-workload-certs\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.159180 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.159158    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-istio-data\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.172342 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.172309    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160fc571-588a-47fd-9acc-f82010429400-istio-podinfo\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.180762 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.180064    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-istio-envoy\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.219978 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.219904    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nx4p\" (UniqueName: \"kubernetes.io/projected/160fc571-588a-47fd-9acc-f82010429400-kube-api-access-8nx4p\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.220209 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.220192    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/160fc571-588a-47fd-9acc-f82010429400-istio-token\") pod \"e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") " pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.282582 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.282525    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:01:59.775057 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.775028    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"]
Nov 02 02:01:59.776676 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:01:59.776644    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160fc571_588a_47fd_9acc_f82010429400.slice/crio-b690303149e35a1cebd34bf1133c28dc6319f5b8bb261fbf947df5ca0c91ddef WatchSource:0}: Error finding container b690303149e35a1cebd34bf1133c28dc6319f5b8bb261fbf947df5ca0c91ddef: Status 404 returned error can't find the container with id b690303149e35a1cebd34bf1133c28dc6319f5b8bb261fbf947df5ca0c91ddef
Nov 02 02:01:59.779444 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.779406    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:01:59.779542 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.779481    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:01:59.779542 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:01:59.779513    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:02:00.494721 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:00.494685    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx" event={"ID":"160fc571-588a-47fd-9acc-f82010429400","Type":"ContainerStarted","Data":"7b07b76c1ddf436f96965edd717d17da1e2483fbb0771470eda6cd9f91b68295"}
Nov 02 02:02:00.494937 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:00.494741    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx" event={"ID":"160fc571-588a-47fd-9acc-f82010429400","Type":"ContainerStarted","Data":"b690303149e35a1cebd34bf1133c28dc6319f5b8bb261fbf947df5ca0c91ddef"}
Nov 02 02:02:00.666330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:00.666265    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx" podStartSLOduration=2.666246395 podStartE2EDuration="2.666246395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:02:00.665410921 +0000 UTC m=+1768.106447416" watchObservedRunningTime="2026-11-02 02:02:00.666246395 +0000 UTC m=+1768.107282893"
Nov 02 02:02:01.282910 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:01.282875    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:02:01.288992 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:01.288968    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:02:01.501993 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:01.501959    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:02:01.503071 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:01.503047    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:02:02.343837 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.343810    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 02:02:02.416704 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.416668    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-istio-data\") pod \"eedec700-8816-4b96-a087-7d0d691123bf\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") "
Nov 02 02:02:02.416887 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.416721    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eedec700-8816-4b96-a087-7d0d691123bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eedec700-8816-4b96-a087-7d0d691123bf-istiod-ca-cert\") pod \"eedec700-8816-4b96-a087-7d0d691123bf\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") "
Nov 02 02:02:02.416887 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.416839    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-credential-socket\") pod \"eedec700-8816-4b96-a087-7d0d691123bf\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") "
Nov 02 02:02:02.417107 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.416902    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eedec700-8816-4b96-a087-7d0d691123bf-istio-token\" (UniqueName: \"kubernetes.io/projected/eedec700-8816-4b96-a087-7d0d691123bf-istio-token\") pod \"eedec700-8816-4b96-a087-7d0d691123bf\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") "
Nov 02 02:02:02.417107 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.416996    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eedec700-8816-4b96-a087-7d0d691123bf-kube-api-access-j9px6\" (UniqueName: \"kubernetes.io/projected/eedec700-8816-4b96-a087-7d0d691123bf-kube-api-access-j9px6\") pod \"eedec700-8816-4b96-a087-7d0d691123bf\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") "
Nov 02 02:02:02.417107 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.417018    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-istio-data" pod "eedec700-8816-4b96-a087-7d0d691123bf" (UID: "eedec700-8816-4b96-a087-7d0d691123bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:02.417337 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.417131    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eedec700-8816-4b96-a087-7d0d691123bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eedec700-8816-4b96-a087-7d0d691123bf-istio-podinfo\") pod \"eedec700-8816-4b96-a087-7d0d691123bf\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") "
Nov 02 02:02:02.417337 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.417163    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-workload-certs\") pod \"eedec700-8816-4b96-a087-7d0d691123bf\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") "
Nov 02 02:02:02.417337 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.417072    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eedec700-8816-4b96-a087-7d0d691123bf-istiod-ca-cert" pod "eedec700-8816-4b96-a087-7d0d691123bf" (UID: "eedec700-8816-4b96-a087-7d0d691123bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:02.417337 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.417160    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-credential-socket" pod "eedec700-8816-4b96-a087-7d0d691123bf" (UID: "eedec700-8816-4b96-a087-7d0d691123bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:02.417337 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.417283    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-workload-socket\") pod \"eedec700-8816-4b96-a087-7d0d691123bf\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") "
Nov 02 02:02:02.417583 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.417366    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-istio-envoy\") pod \"eedec700-8816-4b96-a087-7d0d691123bf\" (UID: \"eedec700-8816-4b96-a087-7d0d691123bf\") "
Nov 02 02:02:02.417583 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.417430    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-workload-certs" pod "eedec700-8816-4b96-a087-7d0d691123bf" (UID: "eedec700-8816-4b96-a087-7d0d691123bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:02.417583 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.417544    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-workload-socket" pod "eedec700-8816-4b96-a087-7d0d691123bf" (UID: "eedec700-8816-4b96-a087-7d0d691123bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:02.418179 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.418157    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:02.418272 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.418185    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eedec700-8816-4b96-a087-7d0d691123bf-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:02.418272 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.418201    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:02.418272 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.418214    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:02.418272 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.418248    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:02.419440 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.419413    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eedec700-8816-4b96-a087-7d0d691123bf-istio-token" pod "eedec700-8816-4b96-a087-7d0d691123bf" (UID: "eedec700-8816-4b96-a087-7d0d691123bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:02.419606 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.419587    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eedec700-8816-4b96-a087-7d0d691123bf-kube-api-access-j9px6" pod "eedec700-8816-4b96-a087-7d0d691123bf" (UID: "eedec700-8816-4b96-a087-7d0d691123bf"). InnerVolumeSpecName "kube-api-access-j9px6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:02.419952 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.419936    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eedec700-8816-4b96-a087-7d0d691123bf-istio-podinfo" pod "eedec700-8816-4b96-a087-7d0d691123bf" (UID: "eedec700-8816-4b96-a087-7d0d691123bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:02:02.420002 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.419958    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-istio-envoy" pod "eedec700-8816-4b96-a087-7d0d691123bf" (UID: "eedec700-8816-4b96-a087-7d0d691123bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:02.510292 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.510186    5875 generic.go:356] "Generic (PLEG): container finished" podID="eedec700-8816-4b96-a087-7d0d691123bf" containerID="e2c1c23aa5ab53e5abc5f336a306437262145fecf70ce32c55303cdfc05a87ee" exitCode=0
Nov 02 02:02:02.510292 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.510246    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"
Nov 02 02:02:02.510292 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.510272    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f" event={"ID":"eedec700-8816-4b96-a087-7d0d691123bf","Type":"ContainerDied","Data":"e2c1c23aa5ab53e5abc5f336a306437262145fecf70ce32c55303cdfc05a87ee"}
Nov 02 02:02:02.510503 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.510332    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f" event={"ID":"eedec700-8816-4b96-a087-7d0d691123bf","Type":"ContainerDied","Data":"da0dcf95c49f05ddd6b2f441f4e84233713ae74e7795a67a23e3f856aa45d7c8"}
Nov 02 02:02:02.510503 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.510361    5875 scope.go:122] "RemoveContainer" containerID="e2c1c23aa5ab53e5abc5f336a306437262145fecf70ce32c55303cdfc05a87ee"
Nov 02 02:02:02.519269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.519241    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eedec700-8816-4b96-a087-7d0d691123bf-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:02.519428 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.519271    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eedec700-8816-4b96-a087-7d0d691123bf-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:02.519428 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.519287    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eedec700-8816-4b96-a087-7d0d691123bf-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:02.519428 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.519303    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9px6\" (UniqueName: \"kubernetes.io/projected/eedec700-8816-4b96-a087-7d0d691123bf-kube-api-access-j9px6\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:02.524085 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.524062    5875 scope.go:122] "RemoveContainer" containerID="7473807fe018380eaa349742a6a3139a3dd4262b48a624125925debf92ce53b4"
Nov 02 02:02:02.534637 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.534580    5875 scope.go:122] "RemoveContainer" containerID="e2c1c23aa5ab53e5abc5f336a306437262145fecf70ce32c55303cdfc05a87ee"
Nov 02 02:02:02.534985 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:02:02.534964    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2c1c23aa5ab53e5abc5f336a306437262145fecf70ce32c55303cdfc05a87ee\": container with ID starting with e2c1c23aa5ab53e5abc5f336a306437262145fecf70ce32c55303cdfc05a87ee not found: ID does not exist" containerID="e2c1c23aa5ab53e5abc5f336a306437262145fecf70ce32c55303cdfc05a87ee"
Nov 02 02:02:02.535081 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.534995    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2c1c23aa5ab53e5abc5f336a306437262145fecf70ce32c55303cdfc05a87ee"} err="failed to get container status \"e2c1c23aa5ab53e5abc5f336a306437262145fecf70ce32c55303cdfc05a87ee\": rpc error: code = NotFound desc = could not find container \"e2c1c23aa5ab53e5abc5f336a306437262145fecf70ce32c55303cdfc05a87ee\": container with ID starting with e2c1c23aa5ab53e5abc5f336a306437262145fecf70ce32c55303cdfc05a87ee not found: ID does not exist"
Nov 02 02:02:02.535081 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.535016    5875 scope.go:122] "RemoveContainer" containerID="7473807fe018380eaa349742a6a3139a3dd4262b48a624125925debf92ce53b4"
Nov 02 02:02:02.535394 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:02:02.535365    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7473807fe018380eaa349742a6a3139a3dd4262b48a624125925debf92ce53b4\": container with ID starting with 7473807fe018380eaa349742a6a3139a3dd4262b48a624125925debf92ce53b4 not found: ID does not exist" containerID="7473807fe018380eaa349742a6a3139a3dd4262b48a624125925debf92ce53b4"
Nov 02 02:02:02.535528 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.535402    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7473807fe018380eaa349742a6a3139a3dd4262b48a624125925debf92ce53b4"} err="failed to get container status \"7473807fe018380eaa349742a6a3139a3dd4262b48a624125925debf92ce53b4\": rpc error: code = NotFound desc = could not find container \"7473807fe018380eaa349742a6a3139a3dd4262b48a624125925debf92ce53b4\": container with ID starting with 7473807fe018380eaa349742a6a3139a3dd4262b48a624125925debf92ce53b4 not found: ID does not exist"
Nov 02 02:02:02.699978 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.699948    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"]
Nov 02 02:02:02.700129 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:02.699995    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-25789ee1-openshift-default-96ffdbf4f-7g97f"]
Nov 02 02:02:04.491663 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:04.491635    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz"
Nov 02 02:02:04.696019 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:04.695985    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eedec700-8816-4b96-a087-7d0d691123bf" path="/var/lib/kubelet/pods/eedec700-8816-4b96-a087-7d0d691123bf/volumes"
Nov 02 02:02:07.342785 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.342746    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv"]
Nov 02 02:02:07.343494 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.343479    5875 cpu_manager.go:414] "containerMap: removing container" podUID="eedec700-8816-4b96-a087-7d0d691123bf" containerName="istio-proxy"
Nov 02 02:02:07.343541 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.343495    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eedec700-8816-4b96-a087-7d0d691123bf" containerName="istio-proxy"
Nov 02 02:02:07.343541 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.343518    5875 cpu_manager.go:414] "containerMap: removing container" podUID="eedec700-8816-4b96-a087-7d0d691123bf" containerName="istio-proxy"
Nov 02 02:02:07.343541 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.343523    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eedec700-8816-4b96-a087-7d0d691123bf" containerName="istio-proxy"
Nov 02 02:02:07.343645 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.343622    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="eedec700-8816-4b96-a087-7d0d691123bf" containerName="istio-proxy"
Nov 02 02:02:07.343645 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.343640    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="eedec700-8816-4b96-a087-7d0d691123bf" containerName="istio-proxy"
Nov 02 02:02:07.349540 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.349518    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv"
Nov 02 02:02:07.457612 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.457572    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-bd3971\""
Nov 02 02:02:07.457991 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.457960    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-bd3971-dockercfg-sjc8p\""
Nov 02 02:02:07.522506 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.522370    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxbtl\" (UniqueName: \"kubernetes.io/projected/40416082-9ca7-4bb0-91f8-534092c7d0c1-kube-api-access-vxbtl\") pod \"payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv\" (UID: \"40416082-9ca7-4bb0-91f8-534092c7d0c1\") " pod="openshift-ingress/payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv"
Nov 02 02:02:07.545359 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.523287    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40416082-9ca7-4bb0-91f8-534092c7d0c1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv\" (UID: \"40416082-9ca7-4bb0-91f8-534092c7d0c1\") " pod="openshift-ingress/payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv"
Nov 02 02:02:07.593414 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.593335    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv"]
Nov 02 02:02:07.637353 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.619488    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8"]
Nov 02 02:02:07.655611 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.655574    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40416082-9ca7-4bb0-91f8-534092c7d0c1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv\" (UID: \"40416082-9ca7-4bb0-91f8-534092c7d0c1\") " pod="openshift-ingress/payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv"
Nov 02 02:02:07.658518 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.658486    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40416082-9ca7-4bb0-91f8-534092c7d0c1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv\" (UID: \"40416082-9ca7-4bb0-91f8-534092c7d0c1\") " pod="openshift-ingress/payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv"
Nov 02 02:02:07.658721 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.658643    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxbtl\" (UniqueName: \"kubernetes.io/projected/40416082-9ca7-4bb0-91f8-534092c7d0c1-kube-api-access-vxbtl\") pod \"payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv\" (UID: \"40416082-9ca7-4bb0-91f8-534092c7d0c1\") " pod="openshift-ingress/payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv"
Nov 02 02:02:07.690747 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.690712    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8"]
Nov 02 02:02:07.691894 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.691872    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8"
Nov 02 02:02:07.733098 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.733050    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxbtl\" (UniqueName: \"kubernetes.io/projected/40416082-9ca7-4bb0-91f8-534092c7d0c1-kube-api-access-vxbtl\") pod \"payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv\" (UID: \"40416082-9ca7-4bb0-91f8-534092c7d0c1\") " pod="openshift-ingress/payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv"
Nov 02 02:02:07.862331 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.862237    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb3138b1-f0f9-4801-9ba7-3ed676116fca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8\" (UID: \"bb3138b1-f0f9-4801-9ba7-3ed676116fca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8"
Nov 02 02:02:07.862486 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.862457    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b5rh\" (UniqueName: \"kubernetes.io/projected/bb3138b1-f0f9-4801-9ba7-3ed676116fca-kube-api-access-6b5rh\") pod \"payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8\" (UID: \"bb3138b1-f0f9-4801-9ba7-3ed676116fca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8"
Nov 02 02:02:07.963877 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.963843    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b5rh\" (UniqueName: \"kubernetes.io/projected/bb3138b1-f0f9-4801-9ba7-3ed676116fca-kube-api-access-6b5rh\") pod \"payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8\" (UID: \"bb3138b1-f0f9-4801-9ba7-3ed676116fca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8"
Nov 02 02:02:07.964060 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.964017    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb3138b1-f0f9-4801-9ba7-3ed676116fca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8\" (UID: \"bb3138b1-f0f9-4801-9ba7-3ed676116fca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8"
Nov 02 02:02:07.964255 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.964216    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv"
Nov 02 02:02:07.964763 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:07.964743    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb3138b1-f0f9-4801-9ba7-3ed676116fca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8\" (UID: \"bb3138b1-f0f9-4801-9ba7-3ed676116fca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8"
Nov 02 02:02:08.028982 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:08.028953    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b5rh\" (UniqueName: \"kubernetes.io/projected/bb3138b1-f0f9-4801-9ba7-3ed676116fca-kube-api-access-6b5rh\") pod \"payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8\" (UID: \"bb3138b1-f0f9-4801-9ba7-3ed676116fca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8"
Nov 02 02:02:08.031442 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:08.031414    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8"
Nov 02 02:02:08.368180 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:08.368137    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv"]
Nov 02 02:02:08.372142 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:02:08.372109    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40416082_9ca7_4bb0_91f8_534092c7d0c1.slice/crio-8e1ea399e52b1bebec0cda1feb7d52614be61a6efcd4e7309eb48d27c2c2fa4a WatchSource:0}: Error finding container 8e1ea399e52b1bebec0cda1feb7d52614be61a6efcd4e7309eb48d27c2c2fa4a: Status 404 returned error can't find the container with id 8e1ea399e52b1bebec0cda1feb7d52614be61a6efcd4e7309eb48d27c2c2fa4a
Nov 02 02:02:08.470276 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:08.470250    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8"]
Nov 02 02:02:08.471320 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:02:08.471291    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3138b1_f0f9_4801_9ba7_3ed676116fca.slice/crio-9b91f7111bbfe225f224141271c605cdaa2d6e1db5abe6ba382744ba86ad8dd3 WatchSource:0}: Error finding container 9b91f7111bbfe225f224141271c605cdaa2d6e1db5abe6ba382744ba86ad8dd3: Status 404 returned error can't find the container with id 9b91f7111bbfe225f224141271c605cdaa2d6e1db5abe6ba382744ba86ad8dd3
Nov 02 02:02:08.730501 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:08.730463    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv" event={"ID":"40416082-9ca7-4bb0-91f8-534092c7d0c1","Type":"ContainerStarted","Data":"1f43212117a3f8c09ef419b1cddede7da589e1b1b02d976e3102e91b089e79c0"}
Nov 02 02:02:08.730708 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:08.730525    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv" event={"ID":"40416082-9ca7-4bb0-91f8-534092c7d0c1","Type":"ContainerStarted","Data":"8e1ea399e52b1bebec0cda1feb7d52614be61a6efcd4e7309eb48d27c2c2fa4a"}
Nov 02 02:02:08.731447 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:08.731421    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv"
Nov 02 02:02:08.735063 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:08.734205    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8" event={"ID":"bb3138b1-f0f9-4801-9ba7-3ed676116fca","Type":"ContainerStarted","Data":"e377507562999a6578c4afaca92c71cf0358bf59f086740b5807c6b935e544c0"}
Nov 02 02:02:08.735063 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:08.734273    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8" event={"ID":"bb3138b1-f0f9-4801-9ba7-3ed676116fca","Type":"ContainerStarted","Data":"9b91f7111bbfe225f224141271c605cdaa2d6e1db5abe6ba382744ba86ad8dd3"}
Nov 02 02:02:08.735362 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:08.735298    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8"
Nov 02 02:02:08.847766 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:08.847711    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv" podStartSLOduration=1.847696858 podStartE2EDuration="1.847696858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:02:08.845372864 +0000 UTC m=+1776.286409361" watchObservedRunningTime="2026-11-02 02:02:08.847696858 +0000 UTC m=+1776.288733355"
Nov 02 02:02:09.021302 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:09.005271    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8" podStartSLOduration=2.005247936 podStartE2EDuration="2.005247936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:02:08.982333675 +0000 UTC m=+1776.423370179" watchObservedRunningTime="2026-11-02 02:02:09.005247936 +0000 UTC m=+1776.446284434"
Nov 02 02:02:11.322273 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:02:11.322210    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:02:14.781536 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:14.781504    5875 generic.go:356] "Generic (PLEG): container finished" podID="a0c5779f-1d48-4749-bcb0-8313b202142d" containerID="547d14704548a99f3eaf569e2fe5da0ee2b16399bd0d050fd3ea68cee2438cdf" exitCode=0
Nov 02 02:02:14.782032 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:14.781586    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t" event={"ID":"a0c5779f-1d48-4749-bcb0-8313b202142d","Type":"ContainerDied","Data":"547d14704548a99f3eaf569e2fe5da0ee2b16399bd0d050fd3ea68cee2438cdf"}
Nov 02 02:02:14.782108 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:14.782063    5875 scope.go:122] "RemoveContainer" containerID="547d14704548a99f3eaf569e2fe5da0ee2b16399bd0d050fd3ea68cee2438cdf"
Nov 02 02:02:14.783835 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:14.783805    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:02:14.783902 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:14.783875    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:02:14.783947 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:14.783908    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:02:15.791171 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:15.791134    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t" event={"ID":"a0c5779f-1d48-4749-bcb0-8313b202142d","Type":"ContainerStarted","Data":"489d1950ec1c74f0377e702f56c7b8ab402593a229a78f42cc4b4c783bcdf856"}
Nov 02 02:02:15.828331 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:15.828297    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw"]
Nov 02 02:02:15.832891 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:15.832868    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw"
Nov 02 02:02:15.886449 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:15.886417    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw"]
Nov 02 02:02:15.892995 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:15.892967    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-11f26c-serving-cert\""
Nov 02 02:02:15.993272 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:15.993235    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d772318-f18c-4308-9e87-182a635baff0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw\" (UID: \"2d772318-f18c-4308-9e87-182a635baff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw"
Nov 02 02:02:15.993550 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:15.993529    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgrjz\" (UniqueName: \"kubernetes.io/projected/2d772318-f18c-4308-9e87-182a635baff0-kube-api-access-tgrjz\") pod \"maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw\" (UID: \"2d772318-f18c-4308-9e87-182a635baff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw"
Nov 02 02:02:15.993640 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:15.993579    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d772318-f18c-4308-9e87-182a635baff0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw\" (UID: \"2d772318-f18c-4308-9e87-182a635baff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw"
Nov 02 02:02:15.993728 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:15.993714    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d772318-f18c-4308-9e87-182a635baff0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw\" (UID: \"2d772318-f18c-4308-9e87-182a635baff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw"
Nov 02 02:02:16.095647 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:16.095563    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgrjz\" (UniqueName: \"kubernetes.io/projected/2d772318-f18c-4308-9e87-182a635baff0-kube-api-access-tgrjz\") pod \"maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw\" (UID: \"2d772318-f18c-4308-9e87-182a635baff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw"
Nov 02 02:02:16.095647 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:16.095607    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d772318-f18c-4308-9e87-182a635baff0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw\" (UID: \"2d772318-f18c-4308-9e87-182a635baff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw"
Nov 02 02:02:16.095884 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:16.095657    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d772318-f18c-4308-9e87-182a635baff0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw\" (UID: \"2d772318-f18c-4308-9e87-182a635baff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw"
Nov 02 02:02:16.095884 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:16.095837    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d772318-f18c-4308-9e87-182a635baff0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw\" (UID: \"2d772318-f18c-4308-9e87-182a635baff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw"
Nov 02 02:02:16.095884 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:02:16.095852    5875 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-serving-cert: secret "maas-api-e2e-shared-b-w3-11f26c-serving-cert" not found
Nov 02 02:02:16.096064 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:02:16.095917    5875 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d772318-f18c-4308-9e87-182a635baff0-maas-api-tls podName:2d772318-f18c-4308-9e87-182a635baff0 nodeName:}" failed. No retries permitted until 2026-11-02 02:02:16.595894855 +0000 UTC m=+1784.036931350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2d772318-f18c-4308-9e87-182a635baff0-maas-api-tls") pod "maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw" (UID: "2d772318-f18c-4308-9e87-182a635baff0") : secret "maas-api-e2e-shared-b-w3-11f26c-serving-cert" not found
Nov 02 02:02:16.096458 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:16.096439    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d772318-f18c-4308-9e87-182a635baff0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw\" (UID: \"2d772318-f18c-4308-9e87-182a635baff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw"
Nov 02 02:02:16.165889 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:16.163375    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d772318-f18c-4308-9e87-182a635baff0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw\" (UID: \"2d772318-f18c-4308-9e87-182a635baff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw"
Nov 02 02:02:16.168246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:16.168199    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgrjz\" (UniqueName: \"kubernetes.io/projected/2d772318-f18c-4308-9e87-182a635baff0-kube-api-access-tgrjz\") pod \"maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw\" (UID: \"2d772318-f18c-4308-9e87-182a635baff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw"
Nov 02 02:02:16.360588 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:16.360485    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:02:16.362212 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:16.362174    5875 patch_prober.go:29] interesting pod/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 02 02:02:16.362414 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:16.362295    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t" podUID="a0c5779f-1d48-4749-bcb0-8313b202142d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 02 02:02:16.626371 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:16.626275    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d772318-f18c-4308-9e87-182a635baff0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw\" (UID: \"2d772318-f18c-4308-9e87-182a635baff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw"
Nov 02 02:02:16.629126 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:16.629100    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d772318-f18c-4308-9e87-182a635baff0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw\" (UID: \"2d772318-f18c-4308-9e87-182a635baff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw"
Nov 02 02:02:16.750233 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:16.750198    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw"
Nov 02 02:02:17.158699 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:17.158664    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw"]
Nov 02 02:02:17.160109 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:02:17.160082    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d772318_f18c_4308_9e87_182a635baff0.slice/crio-9b2e9a0749fdd943541b0df77ff9ce30808a361e370b49947dec6ab2ad46fc24 WatchSource:0}: Error finding container 9b2e9a0749fdd943541b0df77ff9ce30808a361e370b49947dec6ab2ad46fc24: Status 404 returned error can't find the container with id 9b2e9a0749fdd943541b0df77ff9ce30808a361e370b49947dec6ab2ad46fc24
Nov 02 02:02:17.358605 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:17.358571    5875 patch_prober.go:29] interesting pod/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 02 02:02:17.358787 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:17.358626    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t" podUID="a0c5779f-1d48-4749-bcb0-8313b202142d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 02 02:02:17.843273 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:17.843238    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw" event={"ID":"2d772318-f18c-4308-9e87-182a635baff0","Type":"ContainerStarted","Data":"9b2e9a0749fdd943541b0df77ff9ce30808a361e370b49947dec6ab2ad46fc24"}
Nov 02 02:02:18.358517 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:18.358484    5875 patch_prober.go:29] interesting pod/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 02 02:02:18.358958 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:18.358535    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t" podUID="a0c5779f-1d48-4749-bcb0-8313b202142d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 02 02:02:18.852885 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:18.852844    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw" event={"ID":"2d772318-f18c-4308-9e87-182a635baff0","Type":"ContainerStarted","Data":"5ea86199c509316a94721b75bec2085a4fa77ba73b36e17a4c5d6394af7c15f2"}
Nov 02 02:02:18.853068 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:18.852948    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw"
Nov 02 02:02:18.977179 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:18.977114    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw" podStartSLOduration=3.11592365 podStartE2EDuration="3.977094582s" totalImagesPullingTime="861.170932ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:02:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:02:18.975848098 +0000 UTC m=+1786.416884596" watchObservedRunningTime="2026-11-02 02:02:18.977094582 +0000 UTC m=+1786.418131080"
Nov 02 02:02:19.358451 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:19.358420    5875 patch_prober.go:29] interesting pod/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 02 02:02:19.358648 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:19.358486    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t" podUID="a0c5779f-1d48-4749-bcb0-8313b202142d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 02 02:02:19.440043 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:19.440013    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:02:20.358193 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:20.358160    5875 patch_prober.go:29] interesting pod/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 02 02:02:20.358393 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:20.358240    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t" podUID="a0c5779f-1d48-4749-bcb0-8313b202142d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 02 02:02:20.753099 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:20.753071    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv"
Nov 02 02:02:20.753563 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:20.753138    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8"
Nov 02 02:02:21.476034 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:21.475997    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:02:21.478411 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:21.478388    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:02:21.993584 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:21.993553    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"]
Nov 02 02:02:21.997852 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:21.997832    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:02:22.055376 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.055346    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5700d3d1\"/\"kube-root-ca.crt\""
Nov 02 02:02:22.055701 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.055679    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5700d3d1\"/\"openshift-service-ca.crt\""
Nov 02 02:02:22.070558 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.070532    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-tmp-dir\") pod \"e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64\" (UID: \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\") " pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:02:22.070707 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.070569    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-home\") pod \"e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64\" (UID: \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\") " pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:02:22.070707 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.070651    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36d8fa37-0372-40c4-a7d5-fa490c1c460e-tls-certs\") pod \"e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64\" (UID: \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\") " pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:02:22.070882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.070808    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-model-cache\") pod \"e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64\" (UID: \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\") " pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:02:22.071011 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.070993    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-dshm\") pod \"e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64\" (UID: \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\") " pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:02:22.071061 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.071041    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9vn2\" (UniqueName: \"kubernetes.io/projected/36d8fa37-0372-40c4-a7d5-fa490c1c460e-kube-api-access-x9vn2\") pod \"e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64\" (UID: \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\") " pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:02:22.104619 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.104588    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5700d3d1\"/\"default-dockercfg-p4s75\""
Nov 02 02:02:22.104619 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.104587    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5700d3d1\"/\"e2e-lifecycle-model-5700d3d1-kserve-self-signed-certs\""
Nov 02 02:02:22.108080 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.108055    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"]
Nov 02 02:02:22.177354 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.177318    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-dshm\") pod \"e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64\" (UID: \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\") " pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:02:22.177683 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.177655    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9vn2\" (UniqueName: \"kubernetes.io/projected/36d8fa37-0372-40c4-a7d5-fa490c1c460e-kube-api-access-x9vn2\") pod \"e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64\" (UID: \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\") " pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:02:22.177856 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.177841    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-tmp-dir\") pod \"e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64\" (UID: \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\") " pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:02:22.177975 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.177962    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-home\") pod \"e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64\" (UID: \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\") " pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:02:22.178120 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.178103    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36d8fa37-0372-40c4-a7d5-fa490c1c460e-tls-certs\") pod \"e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64\" (UID: \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\") " pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:02:22.178370 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.178347    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-model-cache\") pod \"e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64\" (UID: \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\") " pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:02:22.182914 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.179023    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-model-cache\") pod \"e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64\" (UID: \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\") " pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:02:22.183841 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.183817    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-tmp-dir\") pod \"e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64\" (UID: \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\") " pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:02:22.190156 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.184212    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-home\") pod \"e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64\" (UID: \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\") " pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:02:22.204044 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.204008    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-dshm\") pod \"e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64\" (UID: \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\") " pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:02:22.207098 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.207038    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36d8fa37-0372-40c4-a7d5-fa490c1c460e-tls-certs\") pod \"e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64\" (UID: \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\") " pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:02:22.250135 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.250060    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9vn2\" (UniqueName: \"kubernetes.io/projected/36d8fa37-0372-40c4-a7d5-fa490c1c460e-kube-api-access-x9vn2\") pod \"e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64\" (UID: \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\") " pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:02:22.311252 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.311192    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:02:22.706473 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.706442    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"]
Nov 02 02:02:22.707089 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:02:22.707064    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36d8fa37_0372_40c4_a7d5_fa490c1c460e.slice/crio-5274a062b58bc53320e2a8773917164de7568ec531f907ae33ffc0e04a1013e0 WatchSource:0}: Error finding container 5274a062b58bc53320e2a8773917164de7568ec531f907ae33ffc0e04a1013e0: Status 404 returned error can't find the container with id 5274a062b58bc53320e2a8773917164de7568ec531f907ae33ffc0e04a1013e0
Nov 02 02:02:22.883775 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.883743    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64" event={"ID":"36d8fa37-0372-40c4-a7d5-fa490c1c460e","Type":"ContainerStarted","Data":"6a247511470e3283ae59915a530c076388d2c51af41e4275e564ff31cee58798"}
Nov 02 02:02:22.883940 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:22.883788    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64" event={"ID":"36d8fa37-0372-40c4-a7d5-fa490c1c460e","Type":"ContainerStarted","Data":"5274a062b58bc53320e2a8773917164de7568ec531f907ae33ffc0e04a1013e0"}
Nov 02 02:02:23.000604 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:23.000552    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64" podStartSLOduration=2.000539601 podStartE2EDuration="2.000539601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:02:22.998928352 +0000 UTC m=+1790.439964859" watchObservedRunningTime="2026-11-02 02:02:23.000539601 +0000 UTC m=+1790.441576381"
Nov 02 02:02:23.311491 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:23.311417    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:02:24.866268 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:24.866238    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw"
Nov 02 02:02:32.312107 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:32.312073    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:02:32.325279 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:32.325256    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:02:32.970460 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:32.970435    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:02:40.138318 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:40.134509    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:02:41.327445 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:02:41.327380    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:02:42.101432 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.101398    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"]
Nov 02 02:02:42.109634 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.109606    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.167330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.167299    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-abc85a-openshift-default-dockercfg-dg2jp\""
Nov 02 02:02:42.319245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.249967    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"]
Nov 02 02:02:42.376675 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.376579    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb67a90f-51c6-4440-923d-db0edea22576-istiod-ca-cert\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.377133 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.376683    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-istio-data\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.377133 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.376777    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-credential-socket\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.377133 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.376818    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-workload-certs\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.377133 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.376978    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqbcx\" (UniqueName: \"kubernetes.io/projected/eb67a90f-51c6-4440-923d-db0edea22576-kube-api-access-mqbcx\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.377133 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.377055    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb67a90f-51c6-4440-923d-db0edea22576-istio-podinfo\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.377439 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.377138    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-workload-socket\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.377439 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.377194    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-istio-envoy\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.377439 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.377296    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb67a90f-51c6-4440-923d-db0edea22576-istio-token\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.498012 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.497964    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb67a90f-51c6-4440-923d-db0edea22576-istio-token\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.498423 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.498385    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb67a90f-51c6-4440-923d-db0edea22576-istiod-ca-cert\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.500603 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.500573    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-istio-data\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.500955 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.500847    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-credential-socket\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.501142 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.501127    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-workload-certs\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.501509 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.501489    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqbcx\" (UniqueName: \"kubernetes.io/projected/eb67a90f-51c6-4440-923d-db0edea22576-kube-api-access-mqbcx\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.501679 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.501665    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb67a90f-51c6-4440-923d-db0edea22576-istio-podinfo\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.501777 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.501764    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-workload-socket\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.501910 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.501897    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-istio-envoy\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.505954 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.505679    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-credential-socket\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.507151 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.507128    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb67a90f-51c6-4440-923d-db0edea22576-istiod-ca-cert\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.531966 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.508058    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-workload-certs\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.532290 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.508423    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-workload-socket\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.532478 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.508730    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-istio-data\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.533303 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.533272    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-istio-envoy\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.536918 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.536888    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb67a90f-51c6-4440-923d-db0edea22576-istio-podinfo\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.566668 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.566641    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb67a90f-51c6-4440-923d-db0edea22576-istio-token\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.567357 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.567340    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqbcx\" (UniqueName: \"kubernetes.io/projected/eb67a90f-51c6-4440-923d-db0edea22576-kube-api-access-mqbcx\") pod \"e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") " pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:42.751242 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:42.747698    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:43.231122 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:43.231090    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"]
Nov 02 02:02:43.231728 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:02:43.231700    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb67a90f_51c6_4440_923d_db0edea22576.slice/crio-9af59f28cbb75cb65b522816e0e8388988c093c0daf95d4bde76ec16c4dc06f9 WatchSource:0}: Error finding container 9af59f28cbb75cb65b522816e0e8388988c093c0daf95d4bde76ec16c4dc06f9: Status 404 returned error can't find the container with id 9af59f28cbb75cb65b522816e0e8388988c093c0daf95d4bde76ec16c4dc06f9
Nov 02 02:02:43.234247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:43.234185    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:02:43.234330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:43.234268    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:02:43.234330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:43.234302    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:02:44.139568 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:44.139529    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592" event={"ID":"eb67a90f-51c6-4440-923d-db0edea22576","Type":"ContainerStarted","Data":"218cfc7752e8ecb3b44a5d1566a3b02e44e0596529b3793861d7ecf021601e32"}
Nov 02 02:02:44.139971 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:44.139625    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592" event={"ID":"eb67a90f-51c6-4440-923d-db0edea22576","Type":"ContainerStarted","Data":"9af59f28cbb75cb65b522816e0e8388988c093c0daf95d4bde76ec16c4dc06f9"}
Nov 02 02:02:44.748705 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:44.748673    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:44.754930 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:44.754902    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:44.882581 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:44.882527    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592" podStartSLOduration=2.882511971 podStartE2EDuration="2.882511971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:02:44.322624174 +0000 UTC m=+1811.763660685" watchObservedRunningTime="2026-11-02 02:02:44.882511971 +0000 UTC m=+1812.323548468"
Nov 02 02:02:45.146922 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:45.146818    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:45.147944 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:45.147929    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:02:50.916087 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:50.912614    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb"]
Nov 02 02:02:50.921331 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:50.920887    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb"
Nov 02 02:02:50.968054 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:50.968023    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb"]
Nov 02 02:02:50.973717 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:50.973684    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f75qg\" (UniqueName: \"kubernetes.io/projected/58d02a0f-00d6-4a8f-ac21-b473bd8a440e-kube-api-access-f75qg\") pod \"payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb\" (UID: \"58d02a0f-00d6-4a8f-ac21-b473bd8a440e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb"
Nov 02 02:02:50.973902 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:50.973886    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58d02a0f-00d6-4a8f-ac21-b473bd8a440e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb\" (UID: \"58d02a0f-00d6-4a8f-ac21-b473bd8a440e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb"
Nov 02 02:02:51.021323 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:51.020541    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-abc85a\""
Nov 02 02:02:51.021323 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:51.021280    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-abc85a-dockercfg-vkxl2\""
Nov 02 02:02:51.172297 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:51.112132    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f75qg\" (UniqueName: \"kubernetes.io/projected/58d02a0f-00d6-4a8f-ac21-b473bd8a440e-kube-api-access-f75qg\") pod \"payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb\" (UID: \"58d02a0f-00d6-4a8f-ac21-b473bd8a440e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb"
Nov 02 02:02:51.172297 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:51.112355    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58d02a0f-00d6-4a8f-ac21-b473bd8a440e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb\" (UID: \"58d02a0f-00d6-4a8f-ac21-b473bd8a440e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb"
Nov 02 02:02:51.172297 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:51.113410    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58d02a0f-00d6-4a8f-ac21-b473bd8a440e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb\" (UID: \"58d02a0f-00d6-4a8f-ac21-b473bd8a440e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb"
Nov 02 02:02:51.189320 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:51.175308    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7"]
Nov 02 02:02:51.202938 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:51.201525    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7"]
Nov 02 02:02:51.202938 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:51.201660    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7"
Nov 02 02:02:51.235366 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:51.235210    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f75qg\" (UniqueName: \"kubernetes.io/projected/58d02a0f-00d6-4a8f-ac21-b473bd8a440e-kube-api-access-f75qg\") pod \"payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb\" (UID: \"58d02a0f-00d6-4a8f-ac21-b473bd8a440e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb"
Nov 02 02:02:51.250211 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:51.250176    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb"
Nov 02 02:02:51.327190 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:51.327153    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcx9b\" (UniqueName: \"kubernetes.io/projected/4b5be69e-c305-4530-8322-adf910287bfe-kube-api-access-vcx9b\") pod \"payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7\" (UID: \"4b5be69e-c305-4530-8322-adf910287bfe\") " pod="openshift-ingress/payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7"
Nov 02 02:02:51.327650 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:51.327427    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b5be69e-c305-4530-8322-adf910287bfe-plugins-config-volume\") pod \"payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7\" (UID: \"4b5be69e-c305-4530-8322-adf910287bfe\") " pod="openshift-ingress/payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7"
Nov 02 02:02:51.429726 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:51.429633    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcx9b\" (UniqueName: \"kubernetes.io/projected/4b5be69e-c305-4530-8322-adf910287bfe-kube-api-access-vcx9b\") pod \"payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7\" (UID: \"4b5be69e-c305-4530-8322-adf910287bfe\") " pod="openshift-ingress/payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7"
Nov 02 02:02:51.429901 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:51.429753    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b5be69e-c305-4530-8322-adf910287bfe-plugins-config-volume\") pod \"payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7\" (UID: \"4b5be69e-c305-4530-8322-adf910287bfe\") " pod="openshift-ingress/payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7"
Nov 02 02:02:51.430706 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:51.430679    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b5be69e-c305-4530-8322-adf910287bfe-plugins-config-volume\") pod \"payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7\" (UID: \"4b5be69e-c305-4530-8322-adf910287bfe\") " pod="openshift-ingress/payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7"
Nov 02 02:02:51.491261 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:51.491203    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcx9b\" (UniqueName: \"kubernetes.io/projected/4b5be69e-c305-4530-8322-adf910287bfe-kube-api-access-vcx9b\") pod \"payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7\" (UID: \"4b5be69e-c305-4530-8322-adf910287bfe\") " pod="openshift-ingress/payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7"
Nov 02 02:02:51.601666 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:51.601629    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7"
Nov 02 02:02:51.685512 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:51.684821    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb"]
Nov 02 02:02:51.685748 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:02:51.685461    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d02a0f_00d6_4a8f_ac21_b473bd8a440e.slice/crio-562acb1f8ef2480ccdbe9b86062f22b443cc22976b87d94916962853e38acecc WatchSource:0}: Error finding container 562acb1f8ef2480ccdbe9b86062f22b443cc22976b87d94916962853e38acecc: Status 404 returned error can't find the container with id 562acb1f8ef2480ccdbe9b86062f22b443cc22976b87d94916962853e38acecc
Nov 02 02:02:52.004996 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:52.004952    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7"]
Nov 02 02:02:52.010103 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:02:52.010044    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b5be69e_c305_4530_8322_adf910287bfe.slice/crio-53385cf3735fe116a7fab4f7db8d71c71f5a6a05b665178ee1fb74c98e8991c0 WatchSource:0}: Error finding container 53385cf3735fe116a7fab4f7db8d71c71f5a6a05b665178ee1fb74c98e8991c0: Status 404 returned error can't find the container with id 53385cf3735fe116a7fab4f7db8d71c71f5a6a05b665178ee1fb74c98e8991c0
Nov 02 02:02:52.259063 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:52.258989    5875 generic.go:356] "Generic (PLEG): container finished" podID="d1dafba9-1caf-4355-bfa1-a3b8cea9155e" containerID="a0089823b223237cf52b7b36d05236b80bdae136acb09f33349f664343d476a6" exitCode=0
Nov 02 02:02:52.259271 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:52.259063    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg" event={"ID":"d1dafba9-1caf-4355-bfa1-a3b8cea9155e","Type":"ContainerDied","Data":"a0089823b223237cf52b7b36d05236b80bdae136acb09f33349f664343d476a6"}
Nov 02 02:02:52.259555 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:52.259530    5875 scope.go:122] "RemoveContainer" containerID="a0089823b223237cf52b7b36d05236b80bdae136acb09f33349f664343d476a6"
Nov 02 02:02:52.260956 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:52.260934    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7" event={"ID":"4b5be69e-c305-4530-8322-adf910287bfe","Type":"ContainerStarted","Data":"37cd3f20f89ea14fc2bc6c0c92b208ea764cf3aeda9f5dbe30c308b102381f5a"}
Nov 02 02:02:52.261066 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:52.260989    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7" event={"ID":"4b5be69e-c305-4530-8322-adf910287bfe","Type":"ContainerStarted","Data":"53385cf3735fe116a7fab4f7db8d71c71f5a6a05b665178ee1fb74c98e8991c0"}
Nov 02 02:02:52.261066 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:52.261018    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7"
Nov 02 02:02:52.261368 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:52.261340    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:02:52.261478 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:52.261418    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:02:52.261478 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:52.261455    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:02:52.262636 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:52.262616    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb" event={"ID":"58d02a0f-00d6-4a8f-ac21-b473bd8a440e","Type":"ContainerStarted","Data":"00399438fcfcc16f9183593c759b66bb2ea56ae99cc068a3db2a5a40bb54c271"}
Nov 02 02:02:52.262728 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:52.262654    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb" event={"ID":"58d02a0f-00d6-4a8f-ac21-b473bd8a440e","Type":"ContainerStarted","Data":"562acb1f8ef2480ccdbe9b86062f22b443cc22976b87d94916962853e38acecc"}
Nov 02 02:02:52.262779 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:52.262750    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb"
Nov 02 02:02:52.503504 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:52.503439    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb" podStartSLOduration=2.503424018 podStartE2EDuration="2.503424018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:02:52.499546257 +0000 UTC m=+1819.940582754" watchObservedRunningTime="2026-11-02 02:02:52.503424018 +0000 UTC m=+1819.944460515"
Nov 02 02:02:52.620600 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:52.620480    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7" podStartSLOduration=1.620460639 podStartE2EDuration="1.620460639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:02:52.618529557 +0000 UTC m=+1820.059566055" watchObservedRunningTime="2026-11-02 02:02:52.620460639 +0000 UTC m=+1820.061497138"
Nov 02 02:02:53.272589 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:53.272553    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg" event={"ID":"d1dafba9-1caf-4355-bfa1-a3b8cea9155e","Type":"ContainerStarted","Data":"969612c799fd017191d24f85bce96313458da87cffb21d5fda1afd3d27105b2c"}
Nov 02 02:02:53.655761 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:53.655731    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:02:53.655961 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:53.655937    5875 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 02 02:02:53.656042 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:53.656008    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg" podUID="d1dafba9-1caf-4355-bfa1-a3b8cea9155e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 02 02:02:54.656262 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:54.656198    5875 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 02 02:02:54.656694 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:54.656285    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg" podUID="d1dafba9-1caf-4355-bfa1-a3b8cea9155e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 02 02:02:55.656956 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:55.656921    5875 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 02 02:02:55.657371 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:55.656980    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg" podUID="d1dafba9-1caf-4355-bfa1-a3b8cea9155e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 02 02:02:56.656449 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:56.656412    5875 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 02 02:02:56.656652 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:56.656469    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg" podUID="d1dafba9-1caf-4355-bfa1-a3b8cea9155e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 02 02:02:57.656389 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:57.656357    5875 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 02 02:02:57.656795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:57.656405    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg" podUID="d1dafba9-1caf-4355-bfa1-a3b8cea9155e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 02 02:02:58.655929 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:58.655900    5875 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 02 02:02:58.656101 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:58.655961    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg" podUID="d1dafba9-1caf-4355-bfa1-a3b8cea9155e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 02 02:02:59.661968 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:59.661934    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:02:59.662456 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:59.662423    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:02:59.663171 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:02:59.663155    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:03:03.275629 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:03.275594    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb"
Nov 02 02:03:03.276115 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:03.275672    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7"
Nov 02 02:03:05.534307 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.534272    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"]
Nov 02 02:03:05.539200 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.539178    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.638248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.632523    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-47eb54-openshift-default-dockercfg-w6l26\""
Nov 02 02:03:05.708806 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.708760    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svrm8\" (UniqueName: \"kubernetes.io/projected/0931244f-232a-4588-8b26-eb3b5b5773c2-kube-api-access-svrm8\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.722834 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.709155    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-data\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.821467 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.821376    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-token\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.821630 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.821595    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-workload-socket\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.821718 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.821699    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-envoy\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.821780 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.821741    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-podinfo\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.821848 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.821827    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0931244f-232a-4588-8b26-eb3b5b5773c2-istiod-ca-cert\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.821905 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.821874    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-credential-socket\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.821960 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.821923    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-workload-certs\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.822080 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.801802    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"]
Nov 02 02:03:05.927180 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.924249    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-envoy\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.927180 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.924371    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-podinfo\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.927180 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.924406    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0931244f-232a-4588-8b26-eb3b5b5773c2-istiod-ca-cert\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.927180 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.924545    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-credential-socket\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.927180 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.924708    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-workload-certs\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.927180 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.924804    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svrm8\" (UniqueName: \"kubernetes.io/projected/0931244f-232a-4588-8b26-eb3b5b5773c2-kube-api-access-svrm8\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.927180 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.924929    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-data\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.927180 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.924965    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-token\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.927180 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.925351    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-workload-socket\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.927180 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.925926    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-workload-socket\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.927180 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.926252    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-credential-socket\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.927180 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.926678    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-data\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.927180 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.927026    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-workload-certs\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.928080 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.927551    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0931244f-232a-4588-8b26-eb3b5b5773c2-istiod-ca-cert\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.929987 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.929830    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-envoy\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.931079 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.930822    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-podinfo\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.985997 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.985972    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-token\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:05.986485 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:05.986462    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svrm8\" (UniqueName: \"kubernetes.io/projected/0931244f-232a-4588-8b26-eb3b5b5773c2-kube-api-access-svrm8\") pod \"e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") " pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:06.154156 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:06.154080    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:06.589154 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:06.589122    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"]
Nov 02 02:03:06.590078 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:03:06.590054    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0931244f_232a_4588_8b26_eb3b5b5773c2.slice/crio-d81cbc22e97e5c0cb298b8609ff77e9d112f385aebd60d1ccf9d27036d93ec25 WatchSource:0}: Error finding container d81cbc22e97e5c0cb298b8609ff77e9d112f385aebd60d1ccf9d27036d93ec25: Status 404 returned error can't find the container with id d81cbc22e97e5c0cb298b8609ff77e9d112f385aebd60d1ccf9d27036d93ec25
Nov 02 02:03:06.592907 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:06.592874    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:03:06.592991 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:06.592938    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:03:06.592991 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:06.592971    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:03:07.469380 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:07.469349    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb" event={"ID":"0931244f-232a-4588-8b26-eb3b5b5773c2","Type":"ContainerStarted","Data":"929b255fdf84f63a74a2f0290af1e9bb6a189e5bb6512e7a1665d741a414e661"}
Nov 02 02:03:07.469564 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:07.469404    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb" event={"ID":"0931244f-232a-4588-8b26-eb3b5b5773c2","Type":"ContainerStarted","Data":"d81cbc22e97e5c0cb298b8609ff77e9d112f385aebd60d1ccf9d27036d93ec25"}
Nov 02 02:03:08.155199 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:08.155169    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:08.161681 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:08.161654    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:08.285535 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:08.285484    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb" podStartSLOduration=3.285470525 podStartE2EDuration="3.285470525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:03:07.688032115 +0000 UTC m=+1835.129068614" watchObservedRunningTime="2026-11-02 02:03:08.285470525 +0000 UTC m=+1835.726507022"
Nov 02 02:03:08.476906 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:08.476880    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:08.478015 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:08.477994    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:03:11.339214 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:03:11.339158    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:03:12.622769 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:12.622727    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"]
Nov 02 02:03:12.623676 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:12.623620    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk" podUID="84804eea-7177-41c0-9bd1-63588f418caa" containerName="istio-proxy" containerID="cri-o://8919c2fa00fc8b78ce9a93f2089fae2c5a27aff9b1d650ece888d2c602e46044" gracePeriod=30
Nov 02 02:03:16.769486 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:16.757279    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz"]
Nov 02 02:03:16.798799 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:16.798769    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz"
Nov 02 02:03:16.806801 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:16.806766    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz"]
Nov 02 02:03:16.858542 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:16.858511    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-47eb54\""
Nov 02 02:03:16.884550 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:16.884512    5875 patch_prober.go:29] interesting pod/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:03:16.884861 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:16.884829    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk" podUID="84804eea-7177-41c0-9bd1-63588f418caa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:03:16.915795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:16.896871    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce33230d-5eac-461b-b766-a40de8dda983-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz\" (UID: \"ce33230d-5eac-461b-b766-a40de8dda983\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz"
Nov 02 02:03:16.916352 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:16.916317    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkvx8\" (UniqueName: \"kubernetes.io/projected/ce33230d-5eac-461b-b766-a40de8dda983-kube-api-access-xkvx8\") pod \"payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz\" (UID: \"ce33230d-5eac-461b-b766-a40de8dda983\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz"
Nov 02 02:03:16.963259 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:16.927839    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-47eb54-dockercfg-6fqqs\""
Nov 02 02:03:17.048371 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:17.027801    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce33230d-5eac-461b-b766-a40de8dda983-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz\" (UID: \"ce33230d-5eac-461b-b766-a40de8dda983\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz"
Nov 02 02:03:17.048371 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:17.028041    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkvx8\" (UniqueName: \"kubernetes.io/projected/ce33230d-5eac-461b-b766-a40de8dda983-kube-api-access-xkvx8\") pod \"payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz\" (UID: \"ce33230d-5eac-461b-b766-a40de8dda983\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz"
Nov 02 02:03:17.048371 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:17.029457    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce33230d-5eac-461b-b766-a40de8dda983-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz\" (UID: \"ce33230d-5eac-461b-b766-a40de8dda983\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz"
Nov 02 02:03:17.048371 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:17.034766    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k"]
Nov 02 02:03:17.053682 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:17.053635    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k"]
Nov 02 02:03:17.053832 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:17.053748    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k"
Nov 02 02:03:17.089398 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:17.089372    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkvx8\" (UniqueName: \"kubernetes.io/projected/ce33230d-5eac-461b-b766-a40de8dda983-kube-api-access-xkvx8\") pod \"payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz\" (UID: \"ce33230d-5eac-461b-b766-a40de8dda983\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz"
Nov 02 02:03:17.129059 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:17.129026    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90fc1e25-7db0-4438-b278-07a551ac3314-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k\" (UID: \"90fc1e25-7db0-4438-b278-07a551ac3314\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k"
Nov 02 02:03:17.129211 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:17.129194    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khhc8\" (UniqueName: \"kubernetes.io/projected/90fc1e25-7db0-4438-b278-07a551ac3314-kube-api-access-khhc8\") pod \"payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k\" (UID: \"90fc1e25-7db0-4438-b278-07a551ac3314\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k"
Nov 02 02:03:17.230541 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:17.230503    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90fc1e25-7db0-4438-b278-07a551ac3314-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k\" (UID: \"90fc1e25-7db0-4438-b278-07a551ac3314\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k"
Nov 02 02:03:17.231148 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:17.231129    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90fc1e25-7db0-4438-b278-07a551ac3314-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k\" (UID: \"90fc1e25-7db0-4438-b278-07a551ac3314\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k"
Nov 02 02:03:17.231353 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:17.231338    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khhc8\" (UniqueName: \"kubernetes.io/projected/90fc1e25-7db0-4438-b278-07a551ac3314-kube-api-access-khhc8\") pod \"payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k\" (UID: \"90fc1e25-7db0-4438-b278-07a551ac3314\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k"
Nov 02 02:03:17.265807 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:17.265781    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz"
Nov 02 02:03:17.294259 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:17.294232    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khhc8\" (UniqueName: \"kubernetes.io/projected/90fc1e25-7db0-4438-b278-07a551ac3314-kube-api-access-khhc8\") pod \"payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k\" (UID: \"90fc1e25-7db0-4438-b278-07a551ac3314\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k"
Nov 02 02:03:17.370896 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:17.370661    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k"
Nov 02 02:03:17.844616 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:17.844582    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz"]
Nov 02 02:03:17.886253 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:17.882279    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k"]
Nov 02 02:03:17.979021 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:03:17.978265    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90fc1e25_7db0_4438_b278_07a551ac3314.slice/crio-80fe2d2ea2ef3ba2478310946fde1d5d74e497ebc7791b99715cd20bca56bcae WatchSource:0}: Error finding container 80fe2d2ea2ef3ba2478310946fde1d5d74e497ebc7791b99715cd20bca56bcae: Status 404 returned error can't find the container with id 80fe2d2ea2ef3ba2478310946fde1d5d74e497ebc7791b99715cd20bca56bcae
Nov 02 02:03:18.299023 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.298685    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 02:03:18.328752 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.328727    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-istio-envoy\") pod \"84804eea-7177-41c0-9bd1-63588f418caa\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") "
Nov 02 02:03:18.328914 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.328822    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-credential-socket\") pod \"84804eea-7177-41c0-9bd1-63588f418caa\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") "
Nov 02 02:03:18.328914 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.328867    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84804eea-7177-41c0-9bd1-63588f418caa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84804eea-7177-41c0-9bd1-63588f418caa-istiod-ca-cert\") pod \"84804eea-7177-41c0-9bd1-63588f418caa\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") "
Nov 02 02:03:18.329038 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.328917    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-workload-certs\") pod \"84804eea-7177-41c0-9bd1-63588f418caa\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") "
Nov 02 02:03:18.329038 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.328957    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84804eea-7177-41c0-9bd1-63588f418caa-kube-api-access-54jjw\" (UniqueName: \"kubernetes.io/projected/84804eea-7177-41c0-9bd1-63588f418caa-kube-api-access-54jjw\") pod \"84804eea-7177-41c0-9bd1-63588f418caa\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") "
Nov 02 02:03:18.329150 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.329039    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-workload-socket\") pod \"84804eea-7177-41c0-9bd1-63588f418caa\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") "
Nov 02 02:03:18.329150 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.329126    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-istio-data\") pod \"84804eea-7177-41c0-9bd1-63588f418caa\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") "
Nov 02 02:03:18.329342 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.329316    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-workload-certs" pod "84804eea-7177-41c0-9bd1-63588f418caa" (UID: "84804eea-7177-41c0-9bd1-63588f418caa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:18.329411 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.329377    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-workload-socket" pod "84804eea-7177-41c0-9bd1-63588f418caa" (UID: "84804eea-7177-41c0-9bd1-63588f418caa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:18.329411 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.329393    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-credential-socket" pod "84804eea-7177-41c0-9bd1-63588f418caa" (UID: "84804eea-7177-41c0-9bd1-63588f418caa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:18.329558 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.329534    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-istio-data" pod "84804eea-7177-41c0-9bd1-63588f418caa" (UID: "84804eea-7177-41c0-9bd1-63588f418caa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:18.329632 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.329567    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84804eea-7177-41c0-9bd1-63588f418caa-istio-token\" (UniqueName: \"kubernetes.io/projected/84804eea-7177-41c0-9bd1-63588f418caa-istio-token\") pod \"84804eea-7177-41c0-9bd1-63588f418caa\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") "
Nov 02 02:03:18.329632 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.329596    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84804eea-7177-41c0-9bd1-63588f418caa-istiod-ca-cert" pod "84804eea-7177-41c0-9bd1-63588f418caa" (UID: "84804eea-7177-41c0-9bd1-63588f418caa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:03:18.329743 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.329719    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84804eea-7177-41c0-9bd1-63588f418caa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84804eea-7177-41c0-9bd1-63588f418caa-istio-podinfo\") pod \"84804eea-7177-41c0-9bd1-63588f418caa\" (UID: \"84804eea-7177-41c0-9bd1-63588f418caa\") "
Nov 02 02:03:18.330443 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.330417    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:18.330553 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.330446    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:18.330553 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.330460    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84804eea-7177-41c0-9bd1-63588f418caa-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:18.330553 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.330474    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:18.330553 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.330487    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:18.333730 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.333690    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-istio-envoy" pod "84804eea-7177-41c0-9bd1-63588f418caa" (UID: "84804eea-7177-41c0-9bd1-63588f418caa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:18.333890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.333865    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84804eea-7177-41c0-9bd1-63588f418caa-istio-token" pod "84804eea-7177-41c0-9bd1-63588f418caa" (UID: "84804eea-7177-41c0-9bd1-63588f418caa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:18.333960 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.333888    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84804eea-7177-41c0-9bd1-63588f418caa-kube-api-access-54jjw" pod "84804eea-7177-41c0-9bd1-63588f418caa" (UID: "84804eea-7177-41c0-9bd1-63588f418caa"). InnerVolumeSpecName "kube-api-access-54jjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:18.334042 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.334023    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84804eea-7177-41c0-9bd1-63588f418caa-istio-podinfo" pod "84804eea-7177-41c0-9bd1-63588f418caa" (UID: "84804eea-7177-41c0-9bd1-63588f418caa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:03:18.431883 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.431846    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54jjw\" (UniqueName: \"kubernetes.io/projected/84804eea-7177-41c0-9bd1-63588f418caa-kube-api-access-54jjw\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:18.431883 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.431881    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84804eea-7177-41c0-9bd1-63588f418caa-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:18.432150 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.431895    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84804eea-7177-41c0-9bd1-63588f418caa-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:18.432150 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.431908    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84804eea-7177-41c0-9bd1-63588f418caa-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:18.583778 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.583738    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz" event={"ID":"ce33230d-5eac-461b-b766-a40de8dda983","Type":"ContainerStarted","Data":"c2437684137f1faf93b550e8fcb9bf7a245c72cb8924c3b58dd9bab1a87a457f"}
Nov 02 02:03:18.583978 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.583947    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz"
Nov 02 02:03:18.583978 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.583974    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz" event={"ID":"ce33230d-5eac-461b-b766-a40de8dda983","Type":"ContainerStarted","Data":"e45457eaaea897bfa73a3850f06376191434d10a64bffef004bcfea269b65324"}
Nov 02 02:03:18.585192 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.585164    5875 generic.go:356] "Generic (PLEG): container finished" podID="84804eea-7177-41c0-9bd1-63588f418caa" containerID="8919c2fa00fc8b78ce9a93f2089fae2c5a27aff9b1d650ece888d2c602e46044" exitCode=0
Nov 02 02:03:18.585344 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.585211    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"
Nov 02 02:03:18.585344 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.585286    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk" event={"ID":"84804eea-7177-41c0-9bd1-63588f418caa","Type":"ContainerDied","Data":"8919c2fa00fc8b78ce9a93f2089fae2c5a27aff9b1d650ece888d2c602e46044"}
Nov 02 02:03:18.585488 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.585345    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk" event={"ID":"84804eea-7177-41c0-9bd1-63588f418caa","Type":"ContainerDied","Data":"e47f45192d6bcb58187db915b99b968d4bfe7e12bc8f092a33d11ae8febd0887"}
Nov 02 02:03:18.585488 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.585377    5875 scope.go:122] "RemoveContainer" containerID="8919c2fa00fc8b78ce9a93f2089fae2c5a27aff9b1d650ece888d2c602e46044"
Nov 02 02:03:18.587138 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.587100    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k" event={"ID":"90fc1e25-7db0-4438-b278-07a551ac3314","Type":"ContainerStarted","Data":"3199de6764c8aa2b399dd447fd1e7c8c6f0c5b08f33b69d8224b01e6969178a7"}
Nov 02 02:03:18.587258 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.587144    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k" event={"ID":"90fc1e25-7db0-4438-b278-07a551ac3314","Type":"ContainerStarted","Data":"80fe2d2ea2ef3ba2478310946fde1d5d74e497ebc7791b99715cd20bca56bcae"}
Nov 02 02:03:18.587258 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.587186    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k"
Nov 02 02:03:18.597713 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.597697    5875 scope.go:122] "RemoveContainer" containerID="8919c2fa00fc8b78ce9a93f2089fae2c5a27aff9b1d650ece888d2c602e46044"
Nov 02 02:03:18.598033 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:03:18.598005    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8919c2fa00fc8b78ce9a93f2089fae2c5a27aff9b1d650ece888d2c602e46044\": container with ID starting with 8919c2fa00fc8b78ce9a93f2089fae2c5a27aff9b1d650ece888d2c602e46044 not found: ID does not exist" containerID="8919c2fa00fc8b78ce9a93f2089fae2c5a27aff9b1d650ece888d2c602e46044"
Nov 02 02:03:18.598130 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.598044    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8919c2fa00fc8b78ce9a93f2089fae2c5a27aff9b1d650ece888d2c602e46044"} err="failed to get container status \"8919c2fa00fc8b78ce9a93f2089fae2c5a27aff9b1d650ece888d2c602e46044\": rpc error: code = NotFound desc = could not find container \"8919c2fa00fc8b78ce9a93f2089fae2c5a27aff9b1d650ece888d2c602e46044\": container with ID starting with 8919c2fa00fc8b78ce9a93f2089fae2c5a27aff9b1d650ece888d2c602e46044 not found: ID does not exist"
Nov 02 02:03:18.704565 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.704453    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz" podStartSLOduration=2.7044338679999997 podStartE2EDuration="2.704433868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:03:18.703668064 +0000 UTC m=+1846.144704562" watchObservedRunningTime="2026-11-02 02:03:18.704433868 +0000 UTC m=+1846.145470366"
Nov 02 02:03:18.882920 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.882888    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"]
Nov 02 02:03:18.894295 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.890216    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-5sllk"]
Nov 02 02:03:18.951022 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:18.950965    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k" podStartSLOduration=2.950950703 podStartE2EDuration="2.950950703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:03:18.949300561 +0000 UTC m=+1846.390337073" watchObservedRunningTime="2026-11-02 02:03:18.950950703 +0000 UTC m=+1846.391987200"
Nov 02 02:03:20.697263 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:20.697205    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84804eea-7177-41c0-9bd1-63588f418caa" path="/var/lib/kubelet/pods/84804eea-7177-41c0-9bd1-63588f418caa/volumes"
Nov 02 02:03:22.131740 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.131710    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"]
Nov 02 02:03:22.133780 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.132544    5875 cpu_manager.go:414] "containerMap: removing container" podUID="84804eea-7177-41c0-9bd1-63588f418caa" containerName="istio-proxy"
Nov 02 02:03:22.133780 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.132560    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84804eea-7177-41c0-9bd1-63588f418caa" containerName="istio-proxy"
Nov 02 02:03:22.133780 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.132695    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="84804eea-7177-41c0-9bd1-63588f418caa" containerName="istio-proxy"
Nov 02 02:03:22.136149 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.136132    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.252514 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.252484    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-podinfo\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.252665 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.252642    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-workload-certs\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.252745 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.252734    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-workload-socket\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.252792 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.252754    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-envoy\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.252792 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.252781    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-token\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.252889 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.252876    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c95abfc-9178-4ce3-9e78-cab75960a726-istiod-ca-cert\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.252943 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.252932    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-credential-socket\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.252994 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.252961    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-data\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.253031 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.252994    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7w8c\" (UniqueName: \"kubernetes.io/projected/6c95abfc-9178-4ce3-9e78-cab75960a726-kube-api-access-l7w8c\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.269085 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.266113    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7a24b9-openshift-default-dockercfg-pszfh\""
Nov 02 02:03:22.269593 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.269488    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"]
Nov 02 02:03:22.363009 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.360174    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-workload-certs\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.363009 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.360343    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-workload-socket\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.363009 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.360374    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-envoy\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.363009 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.360419    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-token\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.363009 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.360470    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c95abfc-9178-4ce3-9e78-cab75960a726-istiod-ca-cert\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.363009 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.360552    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-credential-socket\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.363009 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.360598    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-data\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.363009 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.360657    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7w8c\" (UniqueName: \"kubernetes.io/projected/6c95abfc-9178-4ce3-9e78-cab75960a726-kube-api-access-l7w8c\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.363009 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.360717    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-podinfo\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.367958 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.365974    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-workload-certs\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.367958 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.366313    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-workload-socket\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.389508 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.389430    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-data\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.389680 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.389589    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-credential-socket\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.390099 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.390082    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c95abfc-9178-4ce3-9e78-cab75960a726-istiod-ca-cert\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.398966 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.398930    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-podinfo\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.400498 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.400469    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-envoy\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.432157 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.432089    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-token\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.450395 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.450364    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7w8c\" (UniqueName: \"kubernetes.io/projected/6c95abfc-9178-4ce3-9e78-cab75960a726-kube-api-access-l7w8c\") pod \"e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") " pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:22.629208 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.629174    5875 generic.go:356] "Generic (PLEG): container finished" podID="9493769f-3e6b-44f1-aa30-03081e4db485" containerID="581f5dfbafadadcab3095717730bee01dfc9eed93b78c8791fed28e4dc574558" exitCode=0
Nov 02 02:03:22.629392 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.629247    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7" event={"ID":"9493769f-3e6b-44f1-aa30-03081e4db485","Type":"ContainerDied","Data":"581f5dfbafadadcab3095717730bee01dfc9eed93b78c8791fed28e4dc574558"}
Nov 02 02:03:22.629669 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.629655    5875 scope.go:122] "RemoveContainer" containerID="581f5dfbafadadcab3095717730bee01dfc9eed93b78c8791fed28e4dc574558"
Nov 02 02:03:22.631465 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.631420    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:03:22.631620 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.631540    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:03:22.631620 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.631585    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:03:22.750110 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:22.749874    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:23.262622 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.248504    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"]
Nov 02 02:03:23.267391 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.265159    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:03:23.267391 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.265290    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:03:23.267391 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.265338    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:03:23.522293 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.521934    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"]
Nov 02 02:03:23.574074 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.568510    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.583602 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.583566    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"]
Nov 02 02:03:23.661245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.660800    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ldj5\" (UniqueName: \"kubernetes.io/projected/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-kube-api-access-4ldj5\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.661245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.660897    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-envoy\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.661245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.660937    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istiod-ca-cert\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.661245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.660971    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-credential-socket\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.661245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.661080    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-token\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.661245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.661132    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-workload-socket\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.661245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.661180    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-data\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.661245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.661236    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-workload-certs\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.661799 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.661278    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-podinfo\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.694643 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.694606    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-d1405a-openshift-default-dockercfg-zh55s\""
Nov 02 02:03:23.835968 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.811701    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ldj5\" (UniqueName: \"kubernetes.io/projected/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-kube-api-access-4ldj5\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.835968 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.811796    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-envoy\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.835968 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.811834    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istiod-ca-cert\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.835968 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.811868    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-credential-socket\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.835968 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.811964    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-token\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.835968 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.812008    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-workload-socket\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.835968 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.812051    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-data\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.835968 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.812083    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-workload-certs\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.835968 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.812121    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-podinfo\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.835968 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.829191    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istiod-ca-cert\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.869438 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.849939    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-workload-socket\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.869438 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.850170    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-credential-socket\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.869438 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.850414    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-workload-certs\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.869438 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.850622    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-data\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.891328 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.891257    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-envoy\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.920246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.914367    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-podinfo\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.941906 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.941841    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-token\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.942725 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.942680    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ldj5\" (UniqueName: \"kubernetes.io/projected/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-kube-api-access-4ldj5\") pod \"e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") " pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:23.943996 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.943452    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7" event={"ID":"9493769f-3e6b-44f1-aa30-03081e4db485","Type":"ContainerStarted","Data":"e880d4607cd78c7eb96e9c76a71ade86737977b35139bf0db1c1a9a7a07d8eef"}
Nov 02 02:03:23.946432 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.946407    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp" event={"ID":"6c95abfc-9178-4ce3-9e78-cab75960a726","Type":"ContainerStarted","Data":"d9246068f333c1ef64192cb072d014ce4febee610a7cf888bd7c10d508f582aa"}
Nov 02 02:03:23.946547 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:23.946460    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp" event={"ID":"6c95abfc-9178-4ce3-9e78-cab75960a726","Type":"ContainerStarted","Data":"d290222e5de1c4b196b2d474d5cee7f40dac44d868b427ab09d02cba0837b25e"}
Nov 02 02:03:24.034523 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.034486    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf"]
Nov 02 02:03:24.041675 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.041648    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf"
Nov 02 02:03:24.088469 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.088391    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf"]
Nov 02 02:03:24.109300 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.104970    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-bd3971-serving-cert\""
Nov 02 02:03:24.141499 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.141452    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eda11d8-9d48-47e9-b94a-223847be8d73-metrics-certs\") pod \"maas-api-e2e-api-a-bd3971-6848975575-8fwxf\" (UID: \"4eda11d8-9d48-47e9-b94a-223847be8d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf"
Nov 02 02:03:24.141643 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.141536    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qtr4\" (UniqueName: \"kubernetes.io/projected/4eda11d8-9d48-47e9-b94a-223847be8d73-kube-api-access-8qtr4\") pod \"maas-api-e2e-api-a-bd3971-6848975575-8fwxf\" (UID: \"4eda11d8-9d48-47e9-b94a-223847be8d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf"
Nov 02 02:03:24.141756 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.141725    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eda11d8-9d48-47e9-b94a-223847be8d73-maas-api-tls\") pod \"maas-api-e2e-api-a-bd3971-6848975575-8fwxf\" (UID: \"4eda11d8-9d48-47e9-b94a-223847be8d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf"
Nov 02 02:03:24.141969 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.141946    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eda11d8-9d48-47e9-b94a-223847be8d73-openshift-service-ca\") pod \"maas-api-e2e-api-a-bd3971-6848975575-8fwxf\" (UID: \"4eda11d8-9d48-47e9-b94a-223847be8d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf"
Nov 02 02:03:24.192407 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.192347    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp" podStartSLOduration=2.192329041 podStartE2EDuration="2.192329041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:03:24.19010429 +0000 UTC m=+1851.631140787" watchObservedRunningTime="2026-11-02 02:03:24.192329041 +0000 UTC m=+1851.633365539"
Nov 02 02:03:24.232249 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.232188    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:24.243021 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.242994    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eda11d8-9d48-47e9-b94a-223847be8d73-openshift-service-ca\") pod \"maas-api-e2e-api-a-bd3971-6848975575-8fwxf\" (UID: \"4eda11d8-9d48-47e9-b94a-223847be8d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf"
Nov 02 02:03:24.243175 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.243155    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eda11d8-9d48-47e9-b94a-223847be8d73-metrics-certs\") pod \"maas-api-e2e-api-a-bd3971-6848975575-8fwxf\" (UID: \"4eda11d8-9d48-47e9-b94a-223847be8d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf"
Nov 02 02:03:24.243264 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.243247    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qtr4\" (UniqueName: \"kubernetes.io/projected/4eda11d8-9d48-47e9-b94a-223847be8d73-kube-api-access-8qtr4\") pod \"maas-api-e2e-api-a-bd3971-6848975575-8fwxf\" (UID: \"4eda11d8-9d48-47e9-b94a-223847be8d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf"
Nov 02 02:03:24.243339 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.243323    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eda11d8-9d48-47e9-b94a-223847be8d73-maas-api-tls\") pod \"maas-api-e2e-api-a-bd3971-6848975575-8fwxf\" (UID: \"4eda11d8-9d48-47e9-b94a-223847be8d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf"
Nov 02 02:03:24.243584 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:03:24.243565    5875 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-serving-cert: secret "maas-api-e2e-api-a-bd3971-serving-cert" not found
Nov 02 02:03:24.243654 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:03:24.243644    5875 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eda11d8-9d48-47e9-b94a-223847be8d73-maas-api-tls podName:4eda11d8-9d48-47e9-b94a-223847be8d73 nodeName:}" failed. No retries permitted until 2026-11-02 02:03:24.743622387 +0000 UTC m=+1852.184658879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4eda11d8-9d48-47e9-b94a-223847be8d73-maas-api-tls") pod "maas-api-e2e-api-a-bd3971-6848975575-8fwxf" (UID: "4eda11d8-9d48-47e9-b94a-223847be8d73") : secret "maas-api-e2e-api-a-bd3971-serving-cert" not found
Nov 02 02:03:24.243789 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.243768    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eda11d8-9d48-47e9-b94a-223847be8d73-openshift-service-ca\") pod \"maas-api-e2e-api-a-bd3971-6848975575-8fwxf\" (UID: \"4eda11d8-9d48-47e9-b94a-223847be8d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf"
Nov 02 02:03:24.246064 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.246039    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eda11d8-9d48-47e9-b94a-223847be8d73-metrics-certs\") pod \"maas-api-e2e-api-a-bd3971-6848975575-8fwxf\" (UID: \"4eda11d8-9d48-47e9-b94a-223847be8d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf"
Nov 02 02:03:24.333322 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.332026    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qtr4\" (UniqueName: \"kubernetes.io/projected/4eda11d8-9d48-47e9-b94a-223847be8d73-kube-api-access-8qtr4\") pod \"maas-api-e2e-api-a-bd3971-6848975575-8fwxf\" (UID: \"4eda11d8-9d48-47e9-b94a-223847be8d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf"
Nov 02 02:03:24.591021 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.590985    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:03:24.591883 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.591861    5875 patch_prober.go:29] interesting pod/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 02:03:24.592006 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.591906    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7" podUID="9493769f-3e6b-44f1-aa30-03081e4db485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 02:03:24.685574 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.685548    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"]
Nov 02 02:03:24.686551 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:03:24.686517    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ed0d40_2b1a_4e8c_b8c7_abff4f0d972e.slice/crio-e5ae2d9265129749e7d00c445699bf2df12c2a0a26516ee9048bcd317e1f9167 WatchSource:0}: Error finding container e5ae2d9265129749e7d00c445699bf2df12c2a0a26516ee9048bcd317e1f9167: Status 404 returned error can't find the container with id e5ae2d9265129749e7d00c445699bf2df12c2a0a26516ee9048bcd317e1f9167
Nov 02 02:03:24.689309 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.689273    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:03:24.689418 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.689345    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:03:24.689418 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.689376    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:03:24.752451 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.752403    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:24.758008 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.757988    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:24.784976 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.784949    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eda11d8-9d48-47e9-b94a-223847be8d73-maas-api-tls\") pod \"maas-api-e2e-api-a-bd3971-6848975575-8fwxf\" (UID: \"4eda11d8-9d48-47e9-b94a-223847be8d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf"
Nov 02 02:03:24.787607 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.787584    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eda11d8-9d48-47e9-b94a-223847be8d73-maas-api-tls\") pod \"maas-api-e2e-api-a-bd3971-6848975575-8fwxf\" (UID: \"4eda11d8-9d48-47e9-b94a-223847be8d73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf"
Nov 02 02:03:24.957357 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.957330    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf"
Nov 02 02:03:24.957884 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.957853    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9" event={"ID":"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e","Type":"ContainerStarted","Data":"ab0e4330b8ab6fc2f3d8cd2eb1bdf8d62ef21d2d23bd3bcc23dec820a8eddbab"}
Nov 02 02:03:24.958001 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.957926    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9" event={"ID":"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e","Type":"ContainerStarted","Data":"e5ae2d9265129749e7d00c445699bf2df12c2a0a26516ee9048bcd317e1f9167"}
Nov 02 02:03:24.958791 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.958772    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:24.959713 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:24.959695    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:03:25.085416 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:25.085357    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9" podStartSLOduration=2.08533801 podStartE2EDuration="2.08533801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:03:25.081433877 +0000 UTC m=+1852.522470374" watchObservedRunningTime="2026-11-02 02:03:25.08533801 +0000 UTC m=+1852.526374509"
Nov 02 02:03:25.242267 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:25.241880    5875 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 02 02:03:25.242267 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:25.241950    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:25.243056 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:25.241944    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9" podUID="f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 02 02:03:25.378823 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:25.378793    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf"]
Nov 02 02:03:25.392891 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:03:25.392575    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eda11d8_9d48_47e9_b94a_223847be8d73.slice/crio-d04b286005bc55cf8651065936f7d79ed317f9289d36c417b1b66039c157d710 WatchSource:0}: Error finding container d04b286005bc55cf8651065936f7d79ed317f9289d36c417b1b66039c157d710: Status 404 returned error can't find the container with id d04b286005bc55cf8651065936f7d79ed317f9289d36c417b1b66039c157d710
Nov 02 02:03:25.591267 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:25.591158    5875 patch_prober.go:29] interesting pod/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 02:03:25.591267 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:25.591206    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7" podUID="9493769f-3e6b-44f1-aa30-03081e4db485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 02:03:25.967594 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:25.967563    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf" event={"ID":"4eda11d8-9d48-47e9-b94a-223847be8d73","Type":"ContainerStarted","Data":"d04b286005bc55cf8651065936f7d79ed317f9289d36c417b1b66039c157d710"}
Nov 02 02:03:26.238758 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:26.238734    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:26.591134 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:26.591051    5875 patch_prober.go:29] interesting pod/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 02:03:26.591134 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:26.591104    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7" podUID="9493769f-3e6b-44f1-aa30-03081e4db485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 02:03:26.976970 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:26.976934    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf" event={"ID":"4eda11d8-9d48-47e9-b94a-223847be8d73","Type":"ContainerStarted","Data":"eb2739915e8530a67db13dc1e602603a238fb87299d4f365961bd7b69ffdeb72"}
Nov 02 02:03:26.977155 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:26.977031    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf"
Nov 02 02:03:26.977858 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:26.977426    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:26.978073 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:26.978058    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:03:27.097162 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:27.097114    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf" podStartSLOduration=3.264322898 podStartE2EDuration="4.097101737s" totalImagesPullingTime="832.778839ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:03:27.093629888 +0000 UTC m=+1854.534666386" watchObservedRunningTime="2026-11-02 02:03:27.097101737 +0000 UTC m=+1854.538138234"
Nov 02 02:03:27.594247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:27.592663    5875 patch_prober.go:29] interesting pod/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 02:03:27.594247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:27.592726    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7" podUID="9493769f-3e6b-44f1-aa30-03081e4db485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 02:03:28.590484 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:28.590454    5875 patch_prober.go:29] interesting pod/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 02:03:28.590641 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:28.590501    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7" podUID="9493769f-3e6b-44f1-aa30-03081e4db485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 02:03:29.595805 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:29.595768    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:03:29.596202 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:29.596051    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:03:29.596856 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:29.596835    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:03:29.599113 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:29.599094    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k"
Nov 02 02:03:29.599206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:29.599151    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz"
Nov 02 02:03:31.029246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:31.015323    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv"]
Nov 02 02:03:31.055246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:31.041584    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv"
Nov 02 02:03:31.089253 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:31.088563    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv"]
Nov 02 02:03:31.115244 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:31.098619    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7a24b9\""
Nov 02 02:03:31.116310 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:31.100192    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7a24b9-dockercfg-j84j7\""
Nov 02 02:03:31.174636 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:31.174605    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729"]
Nov 02 02:03:31.181038 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:31.180998    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729"
Nov 02 02:03:31.189283 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:31.188576    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92ksk\" (UniqueName: \"kubernetes.io/projected/f6d65404-5fe2-41e1-b878-634a13c47eb8-kube-api-access-92ksk\") pod \"payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv\" (UID: \"f6d65404-5fe2-41e1-b878-634a13c47eb8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv"
Nov 02 02:03:31.196306 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:31.189774    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6d65404-5fe2-41e1-b878-634a13c47eb8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv\" (UID: \"f6d65404-5fe2-41e1-b878-634a13c47eb8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv"
Nov 02 02:03:31.236910 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:31.236882    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729"]
Nov 02 02:03:31.291709 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:31.291633    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6d65404-5fe2-41e1-b878-634a13c47eb8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv\" (UID: \"f6d65404-5fe2-41e1-b878-634a13c47eb8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv"
Nov 02 02:03:31.291709 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:31.291694    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a587ba49-1db6-49fb-9b21-9a00b9eb504f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729\" (UID: \"a587ba49-1db6-49fb-9b21-9a00b9eb504f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729"
Nov 02 02:03:31.291954 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:31.291928    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92ksk\" (UniqueName: \"kubernetes.io/projected/f6d65404-5fe2-41e1-b878-634a13c47eb8-kube-api-access-92ksk\") pod \"payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv\" (UID: \"f6d65404-5fe2-41e1-b878-634a13c47eb8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv"
Nov 02 02:03:31.292031 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:31.292015    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swzrx\" (UniqueName: \"kubernetes.io/projected/a587ba49-1db6-49fb-9b21-9a00b9eb504f-kube-api-access-swzrx\") pod \"payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729\" (UID: \"a587ba49-1db6-49fb-9b21-9a00b9eb504f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729"
Nov 02 02:03:31.292340 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:31.292324    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6d65404-5fe2-41e1-b878-634a13c47eb8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv\" (UID: \"f6d65404-5fe2-41e1-b878-634a13c47eb8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv"
Nov 02 02:03:31.352673 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:31.352648    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92ksk\" (UniqueName: \"kubernetes.io/projected/f6d65404-5fe2-41e1-b878-634a13c47eb8-kube-api-access-92ksk\") pod \"payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv\" (UID: \"f6d65404-5fe2-41e1-b878-634a13c47eb8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv"
Nov 02 02:03:31.393503 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:31.393473    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swzrx\" (UniqueName: \"kubernetes.io/projected/a587ba49-1db6-49fb-9b21-9a00b9eb504f-kube-api-access-swzrx\") pod \"payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729\" (UID: \"a587ba49-1db6-49fb-9b21-9a00b9eb504f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729"
Nov 02 02:03:31.393640 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:31.393617    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a587ba49-1db6-49fb-9b21-9a00b9eb504f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729\" (UID: \"a587ba49-1db6-49fb-9b21-9a00b9eb504f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729"
Nov 02 02:03:31.394316 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:31.394294    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a587ba49-1db6-49fb-9b21-9a00b9eb504f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729\" (UID: \"a587ba49-1db6-49fb-9b21-9a00b9eb504f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729"
Nov 02 02:03:31.436068 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:31.436043    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv"
Nov 02 02:03:31.455745 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:31.455719    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swzrx\" (UniqueName: \"kubernetes.io/projected/a587ba49-1db6-49fb-9b21-9a00b9eb504f-kube-api-access-swzrx\") pod \"payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729\" (UID: \"a587ba49-1db6-49fb-9b21-9a00b9eb504f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729"
Nov 02 02:03:31.523537 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:31.523483    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729"
Nov 02 02:03:31.820026 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:31.819993    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv"]
Nov 02 02:03:31.821995 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:03:31.821967    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d65404_5fe2_41e1_b878_634a13c47eb8.slice/crio-41f9ec88887ec464d913cbd3ea4cf7ffb84335aec1756fa60f5d9976dae24a05 WatchSource:0}: Error finding container 41f9ec88887ec464d913cbd3ea4cf7ffb84335aec1756fa60f5d9976dae24a05: Status 404 returned error can't find the container with id 41f9ec88887ec464d913cbd3ea4cf7ffb84335aec1756fa60f5d9976dae24a05
Nov 02 02:03:31.920306 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:31.920277    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729"]
Nov 02 02:03:31.920652 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:03:31.920630    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda587ba49_1db6_49fb_9b21_9a00b9eb504f.slice/crio-7ae036d3b509905e2a08ca29f55f59ea39ced3aecbd11f605967dafbe518cbca WatchSource:0}: Error finding container 7ae036d3b509905e2a08ca29f55f59ea39ced3aecbd11f605967dafbe518cbca: Status 404 returned error can't find the container with id 7ae036d3b509905e2a08ca29f55f59ea39ced3aecbd11f605967dafbe518cbca
Nov 02 02:03:32.088507 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:32.088466    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729" event={"ID":"a587ba49-1db6-49fb-9b21-9a00b9eb504f","Type":"ContainerStarted","Data":"a8e1fd997e7a92ee40a951feb7c33acf8a2952a4042079dce901cc4451481919"}
Nov 02 02:03:32.088937 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:32.088540    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729" event={"ID":"a587ba49-1db6-49fb-9b21-9a00b9eb504f","Type":"ContainerStarted","Data":"7ae036d3b509905e2a08ca29f55f59ea39ced3aecbd11f605967dafbe518cbca"}
Nov 02 02:03:32.088937 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:32.088696    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729"
Nov 02 02:03:32.090288 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:32.090262    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv" event={"ID":"f6d65404-5fe2-41e1-b878-634a13c47eb8","Type":"ContainerStarted","Data":"a85a74d16a85bc6c18cc1125c01b49356238376a4f4953c844655838b7764f2d"}
Nov 02 02:03:32.090409 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:32.090300    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv" event={"ID":"f6d65404-5fe2-41e1-b878-634a13c47eb8","Type":"ContainerStarted","Data":"41f9ec88887ec464d913cbd3ea4cf7ffb84335aec1756fa60f5d9976dae24a05"}
Nov 02 02:03:32.090409 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:32.090319    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv"
Nov 02 02:03:32.206130 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:32.206091    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729" podStartSLOduration=1.206078952 podStartE2EDuration="1.206078952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:03:32.204119492 +0000 UTC m=+1859.645155989" watchObservedRunningTime="2026-11-02 02:03:32.206078952 +0000 UTC m=+1859.647115449"
Nov 02 02:03:32.340460 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:32.321550    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv" podStartSLOduration=2.32152881 podStartE2EDuration="2.32152881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:03:32.320597288 +0000 UTC m=+1859.761633797" watchObservedRunningTime="2026-11-02 02:03:32.32152881 +0000 UTC m=+1859.762565308"
Nov 02 02:03:32.990097 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:32.990070    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf"
Nov 02 02:03:38.055464 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:38.055409    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z"]
Nov 02 02:03:38.071244 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:38.067473    5875 scope.go:122] "RemoveContainer" containerID="89d635edc9dec71bdc170eb17ba2ebca30e58b548feb7bd22273199a068a1f9a"
Nov 02 02:03:38.071244 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:38.069380    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z"
Nov 02 02:03:38.087297 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:38.087278    5875 scope.go:122] "RemoveContainer" containerID="f899001475e40df82e526f4f7f66c6e3617edb0d191faa7bdadfdb85ecbc1c3d"
Nov 02 02:03:38.097138 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:38.097119    5875 scope.go:122] "RemoveContainer" containerID="b2be452aa679989f7fce969b4082e74818e7cbf4ec022e1497a7f4f87250b3c6"
Nov 02 02:03:38.106009 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:38.105986    5875 scope.go:122] "RemoveContainer" containerID="a876823ec2d33fcb50be5b3c888236371e706e83dd4151eea39d0f4e0e8c6317"
Nov 02 02:03:38.111185 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:38.111160    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32512901-d0e2-4df1-bab5-571d7434ed06-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z\" (UID: \"32512901-d0e2-4df1-bab5-571d7434ed06\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z"
Nov 02 02:03:38.111598 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:38.111573    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr766\" (UniqueName: \"kubernetes.io/projected/32512901-d0e2-4df1-bab5-571d7434ed06-kube-api-access-rr766\") pod \"payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z\" (UID: \"32512901-d0e2-4df1-bab5-571d7434ed06\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z"
Nov 02 02:03:38.119395 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:38.119361    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z"]
Nov 02 02:03:38.164596 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:38.164558    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-d1405a\""
Nov 02 02:03:38.164753 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:38.164715    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-d1405a-dockercfg-b2l9v\""
Nov 02 02:03:38.217248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:38.213657    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr766\" (UniqueName: \"kubernetes.io/projected/32512901-d0e2-4df1-bab5-571d7434ed06-kube-api-access-rr766\") pod \"payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z\" (UID: \"32512901-d0e2-4df1-bab5-571d7434ed06\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z"
Nov 02 02:03:38.217248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:38.213907    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32512901-d0e2-4df1-bab5-571d7434ed06-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z\" (UID: \"32512901-d0e2-4df1-bab5-571d7434ed06\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z"
Nov 02 02:03:38.217248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:38.214928    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32512901-d0e2-4df1-bab5-571d7434ed06-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z\" (UID: \"32512901-d0e2-4df1-bab5-571d7434ed06\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z"
Nov 02 02:03:38.296244 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:38.289490    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr766\" (UniqueName: \"kubernetes.io/projected/32512901-d0e2-4df1-bab5-571d7434ed06-kube-api-access-rr766\") pod \"payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z\" (UID: \"32512901-d0e2-4df1-bab5-571d7434ed06\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z"
Nov 02 02:03:38.392279 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:38.392016    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z"
Nov 02 02:03:38.786065 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:38.786036    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z"]
Nov 02 02:03:38.786820 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:03:38.786792    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32512901_d0e2_4df1_bab5_571d7434ed06.slice/crio-44a131c0fa7357c94892a14daefc608e292597d037602d697c979451b1030934 WatchSource:0}: Error finding container 44a131c0fa7357c94892a14daefc608e292597d037602d697c979451b1030934: Status 404 returned error can't find the container with id 44a131c0fa7357c94892a14daefc608e292597d037602d697c979451b1030934
Nov 02 02:03:39.166697 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:39.166609    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z" event={"ID":"32512901-d0e2-4df1-bab5-571d7434ed06","Type":"ContainerStarted","Data":"84cd707cee34835bf493bed0395d3da93a340122bb2ca137f21b675dfb5da182"}
Nov 02 02:03:39.166697 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:39.166697    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z"
Nov 02 02:03:39.167137 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:39.166719    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z" event={"ID":"32512901-d0e2-4df1-bab5-571d7434ed06","Type":"ContainerStarted","Data":"44a131c0fa7357c94892a14daefc608e292597d037602d697c979451b1030934"}
Nov 02 02:03:39.285618 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:39.285559    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z" podStartSLOduration=1.285546225 podStartE2EDuration="1.285546225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:03:39.281428031 +0000 UTC m=+1866.722464525" watchObservedRunningTime="2026-11-02 02:03:39.285546225 +0000 UTC m=+1866.726582721"
Nov 02 02:03:41.322198 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:03:41.322160    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:03:43.102042 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:43.102014    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv"
Nov 02 02:03:43.102507 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:43.102079    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729"
Nov 02 02:03:50.206587 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.206562    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z"
Nov 02 02:03:50.605648 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.605567    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"]
Nov 02 02:03:50.611595 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.611572    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:50.671256 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.667186    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-aa1597-openshift-default-dockercfg-mzqzn\""
Nov 02 02:03:50.857601 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.839324    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fda26851-8e30-489a-896d-73d9bcdd783f-istiod-ca-cert\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:50.857601 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.839530    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fda26851-8e30-489a-896d-73d9bcdd783f-istio-token\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:50.857601 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.839588    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-istio-data\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:50.857601 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.839639    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-istio-envoy\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:50.857601 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.839671    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-credential-socket\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:50.857601 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.839747    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knzhv\" (UniqueName: \"kubernetes.io/projected/fda26851-8e30-489a-896d-73d9bcdd783f-kube-api-access-knzhv\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:50.857601 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.839784    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-workload-socket\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:50.857601 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.839827    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-workload-certs\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:50.857601 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.839852    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fda26851-8e30-489a-896d-73d9bcdd783f-istio-podinfo\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:50.857601 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.855446    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"]
Nov 02 02:03:50.983066 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.983030    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fda26851-8e30-489a-896d-73d9bcdd783f-istio-token\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:50.983398 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.983375    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-istio-data\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:50.983570 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.983555    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-istio-envoy\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:50.983742 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.983720    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-credential-socket\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:50.983973 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.983953    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knzhv\" (UniqueName: \"kubernetes.io/projected/fda26851-8e30-489a-896d-73d9bcdd783f-kube-api-access-knzhv\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:50.984115 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.984100    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-workload-socket\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:50.984273 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.984256    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-workload-certs\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:50.992711 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.985136    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fda26851-8e30-489a-896d-73d9bcdd783f-istio-podinfo\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:50.993052 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.993026    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fda26851-8e30-489a-896d-73d9bcdd783f-istiod-ca-cert\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:50.994183 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.994155    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fda26851-8e30-489a-896d-73d9bcdd783f-istiod-ca-cert\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:50.999181 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.995843    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-istio-data\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:50.999419 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.996101    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-workload-socket\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:50.999522 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.996376    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-credential-socket\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:50.999607 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:50.997791    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-workload-certs\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:51.008929 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:51.008900    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-istio-envoy\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:51.010207 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:51.010149    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fda26851-8e30-489a-896d-73d9bcdd783f-istio-podinfo\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:51.073001 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:51.072966    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fda26851-8e30-489a-896d-73d9bcdd783f-istio-token\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:51.073256 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:51.073050    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knzhv\" (UniqueName: \"kubernetes.io/projected/fda26851-8e30-489a-896d-73d9bcdd783f-kube-api-access-knzhv\") pod \"e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") " pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:51.242788 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:51.242756    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:51.732878 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:03:51.732847    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda26851_8e30_489a_896d_73d9bcdd783f.slice/crio-ad675c0ec177379d8cafcda3cd81466e0634870b775feb075307a13998036b35 WatchSource:0}: Error finding container ad675c0ec177379d8cafcda3cd81466e0634870b775feb075307a13998036b35: Status 404 returned error can't find the container with id ad675c0ec177379d8cafcda3cd81466e0634870b775feb075307a13998036b35
Nov 02 02:03:51.734216 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:51.734186    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"]
Nov 02 02:03:51.736178 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:51.736135    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:03:51.736295 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:51.736244    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:03:51.736295 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:51.736286    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:03:52.368830 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:52.368788    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk" event={"ID":"fda26851-8e30-489a-896d-73d9bcdd783f","Type":"ContainerStarted","Data":"4eff3eed579d2fc8030603ab4af2ef048a5dac16cfe3c875cdf4bf341d9b5d82"}
Nov 02 02:03:52.369377 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:52.368846    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk" event={"ID":"fda26851-8e30-489a-896d-73d9bcdd783f","Type":"ContainerStarted","Data":"ad675c0ec177379d8cafcda3cd81466e0634870b775feb075307a13998036b35"}
Nov 02 02:03:52.489288 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:52.489209    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk" podStartSLOduration=2.489195258 podStartE2EDuration="2.489195258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:03:52.488153213 +0000 UTC m=+1879.929189711" watchObservedRunningTime="2026-11-02 02:03:52.489195258 +0000 UTC m=+1879.930231754"
Nov 02 02:03:53.244136 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:53.244097    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:53.251289 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:53.251259    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:53.377627 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:53.377546    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:53.378589 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:53.378569    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:03:54.386543 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:54.386451    5875 generic.go:356] "Generic (PLEG): container finished" podID="4650bde4-c321-40aa-bc8e-af7bc0dd3311" containerID="d3e2a3f5397eefe994e1bbd3e203ce3edb73f56cff8cf6f0757f72524a2ffc58" exitCode=0
Nov 02 02:03:54.386543 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:54.386515    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92" event={"ID":"4650bde4-c321-40aa-bc8e-af7bc0dd3311","Type":"ContainerDied","Data":"d3e2a3f5397eefe994e1bbd3e203ce3edb73f56cff8cf6f0757f72524a2ffc58"}
Nov 02 02:03:54.387286 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:54.387269    5875 scope.go:122] "RemoveContainer" containerID="d3e2a3f5397eefe994e1bbd3e203ce3edb73f56cff8cf6f0757f72524a2ffc58"
Nov 02 02:03:54.388906 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:54.388874    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:03:54.389012 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:54.388956    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:03:54.389078 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:54.389008    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:03:55.396533 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:55.396496    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92" event={"ID":"4650bde4-c321-40aa-bc8e-af7bc0dd3311","Type":"ContainerStarted","Data":"b446b7bc1e91f16ddbf4398719abaf05c97cec9a6a4b04f69eeebd8b0bd4ee82"}
Nov 02 02:03:55.532654 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:55.532612    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:03:55.533040 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:55.532943    5875 patch_prober.go:29] interesting pod/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 02 02:03:55.533265 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:55.533236    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92" podUID="4650bde4-c321-40aa-bc8e-af7bc0dd3311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 02 02:03:56.524754 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:56.524718    5875 patch_prober.go:29] interesting pod/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 02 02:03:56.525203 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:56.524773    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92" podUID="4650bde4-c321-40aa-bc8e-af7bc0dd3311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 02 02:03:57.524116 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:57.524062    5875 patch_prober.go:29] interesting pod/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 02 02:03:57.524326 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:57.524135    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92" podUID="4650bde4-c321-40aa-bc8e-af7bc0dd3311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 02 02:03:58.525007 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:58.524966    5875 patch_prober.go:29] interesting pod/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 02 02:03:58.525475 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:58.525028    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92" podUID="4650bde4-c321-40aa-bc8e-af7bc0dd3311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 02 02:03:59.524186 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:59.524152    5875 patch_prober.go:29] interesting pod/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 02 02:03:59.524424 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:03:59.524208    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92" podUID="4650bde4-c321-40aa-bc8e-af7bc0dd3311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 02 02:04:00.048450 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:00.048418    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d8d7057-w7tml"]
Nov 02 02:04:00.122249 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:00.121583    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d8d7057-w7tml"]
Nov 02 02:04:00.524153 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:00.524109    5875 patch_prober.go:29] interesting pod/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 02 02:04:00.524394 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:00.524168    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92" podUID="4650bde4-c321-40aa-bc8e-af7bc0dd3311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 02 02:04:00.697777 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:00.697743    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bdf2f06-8069-472f-81f1-7def8ca7cb27" path="/var/lib/kubelet/pods/2bdf2f06-8069-472f-81f1-7def8ca7cb27/volumes"
Nov 02 02:04:01.256249 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:01.242198    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck"]
Nov 02 02:04:01.299248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:01.294991    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck"
Nov 02 02:04:01.318248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:01.306162    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck"]
Nov 02 02:04:01.352245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:01.343426    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64zvf\" (UniqueName: \"kubernetes.io/projected/e52a668a-65c7-4961-9af7-8f176d9f2c2b-kube-api-access-64zvf\") pod \"payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck\" (UID: \"e52a668a-65c7-4961-9af7-8f176d9f2c2b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck"
Nov 02 02:04:01.352787 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:01.352754    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52a668a-65c7-4961-9af7-8f176d9f2c2b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck\" (UID: \"e52a668a-65c7-4961-9af7-8f176d9f2c2b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck"
Nov 02 02:04:01.369242 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:01.350681    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-aa1597-dockercfg-fjdrk\""
Nov 02 02:04:01.369482 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:01.350538    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-aa1597\""
Nov 02 02:04:01.459442 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:01.459408    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64zvf\" (UniqueName: \"kubernetes.io/projected/e52a668a-65c7-4961-9af7-8f176d9f2c2b-kube-api-access-64zvf\") pod \"payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck\" (UID: \"e52a668a-65c7-4961-9af7-8f176d9f2c2b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck"
Nov 02 02:04:01.459648 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:01.459523    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52a668a-65c7-4961-9af7-8f176d9f2c2b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck\" (UID: \"e52a668a-65c7-4961-9af7-8f176d9f2c2b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck"
Nov 02 02:04:01.460208 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:01.460186    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52a668a-65c7-4961-9af7-8f176d9f2c2b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck\" (UID: \"e52a668a-65c7-4961-9af7-8f176d9f2c2b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck"
Nov 02 02:04:01.520557 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:01.520490    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64zvf\" (UniqueName: \"kubernetes.io/projected/e52a668a-65c7-4961-9af7-8f176d9f2c2b-kube-api-access-64zvf\") pod \"payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck\" (UID: \"e52a668a-65c7-4961-9af7-8f176d9f2c2b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck"
Nov 02 02:04:01.529020 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:01.528991    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:04:01.529455 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:01.529437    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:04:01.530032 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:01.530015    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:04:01.733885 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:01.733858    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck"
Nov 02 02:04:02.118543 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:02.118507    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck"]
Nov 02 02:04:02.119197 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:04:02.119172    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode52a668a_65c7_4961_9af7_8f176d9f2c2b.slice/crio-e91702ecebbfc894608a2e3ef6722216a36b3e84ba837a8baab4aa99a3175b07 WatchSource:0}: Error finding container e91702ecebbfc894608a2e3ef6722216a36b3e84ba837a8baab4aa99a3175b07: Status 404 returned error can't find the container with id e91702ecebbfc894608a2e3ef6722216a36b3e84ba837a8baab4aa99a3175b07
Nov 02 02:04:02.479793 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:02.479758    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck" event={"ID":"e52a668a-65c7-4961-9af7-8f176d9f2c2b","Type":"ContainerStarted","Data":"7d6a1f1320e2d0575b2a84eeaca5c016624526584dc39ac494df74af04df3996"}
Nov 02 02:04:02.480211 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:02.479817    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck" event={"ID":"e52a668a-65c7-4961-9af7-8f176d9f2c2b","Type":"ContainerStarted","Data":"e91702ecebbfc894608a2e3ef6722216a36b3e84ba837a8baab4aa99a3175b07"}
Nov 02 02:04:02.480211 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:02.480042    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck"
Nov 02 02:04:02.598072 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:02.598016    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck" podStartSLOduration=1.597997025 podStartE2EDuration="1.597997025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:04:02.595023928 +0000 UTC m=+1890.036060428" watchObservedRunningTime="2026-11-02 02:04:02.597997025 +0000 UTC m=+1890.039033522"
Nov 02 02:04:07.023103 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:07.023069    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:04:10.378149 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:10.378071    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv"]
Nov 02 02:04:10.384137 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:10.384113    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv"
Nov 02 02:04:10.442707 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:10.442672    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv"]
Nov 02 02:04:10.447107 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:10.447072    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc421d13-d643-4058-bcfa-1a44a56365eb-maas-api-tls\") pod \"maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv\" (UID: \"fc421d13-d643-4058-bcfa-1a44a56365eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv"
Nov 02 02:04:10.447286 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:10.447170    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj5h9\" (UniqueName: \"kubernetes.io/projected/fc421d13-d643-4058-bcfa-1a44a56365eb-kube-api-access-xj5h9\") pod \"maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv\" (UID: \"fc421d13-d643-4058-bcfa-1a44a56365eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv"
Nov 02 02:04:10.447432 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:10.447380    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc421d13-d643-4058-bcfa-1a44a56365eb-openshift-service-ca\") pod \"maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv\" (UID: \"fc421d13-d643-4058-bcfa-1a44a56365eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv"
Nov 02 02:04:10.447679 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:10.447645    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc421d13-d643-4058-bcfa-1a44a56365eb-metrics-certs\") pod \"maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv\" (UID: \"fc421d13-d643-4058-bcfa-1a44a56365eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv"
Nov 02 02:04:10.486368 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:10.486342    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-abc85a-serving-cert\""
Nov 02 02:04:10.549378 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:10.549326    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc421d13-d643-4058-bcfa-1a44a56365eb-openshift-service-ca\") pod \"maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv\" (UID: \"fc421d13-d643-4058-bcfa-1a44a56365eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv"
Nov 02 02:04:10.549620 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:10.549598    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc421d13-d643-4058-bcfa-1a44a56365eb-metrics-certs\") pod \"maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv\" (UID: \"fc421d13-d643-4058-bcfa-1a44a56365eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv"
Nov 02 02:04:10.549715 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:10.549659    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc421d13-d643-4058-bcfa-1a44a56365eb-maas-api-tls\") pod \"maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv\" (UID: \"fc421d13-d643-4058-bcfa-1a44a56365eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv"
Nov 02 02:04:10.549715 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:10.549695    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj5h9\" (UniqueName: \"kubernetes.io/projected/fc421d13-d643-4058-bcfa-1a44a56365eb-kube-api-access-xj5h9\") pod \"maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv\" (UID: \"fc421d13-d643-4058-bcfa-1a44a56365eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv"
Nov 02 02:04:10.552331 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:10.550850    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc421d13-d643-4058-bcfa-1a44a56365eb-openshift-service-ca\") pod \"maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv\" (UID: \"fc421d13-d643-4058-bcfa-1a44a56365eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv"
Nov 02 02:04:10.552331 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:04:10.551014    5875 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-serving-cert: secret "maas-api-e2e-api-a-abc85a-serving-cert" not found
Nov 02 02:04:10.552331 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:04:10.551091    5875 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc421d13-d643-4058-bcfa-1a44a56365eb-maas-api-tls podName:fc421d13-d643-4058-bcfa-1a44a56365eb nodeName:}" failed. No retries permitted until 2026-11-02 02:04:11.05107048 +0000 UTC m=+1898.492106971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fc421d13-d643-4058-bcfa-1a44a56365eb-maas-api-tls") pod "maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv" (UID: "fc421d13-d643-4058-bcfa-1a44a56365eb") : secret "maas-api-e2e-api-a-abc85a-serving-cert" not found
Nov 02 02:04:10.557683 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:10.557657    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc421d13-d643-4058-bcfa-1a44a56365eb-metrics-certs\") pod \"maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv\" (UID: \"fc421d13-d643-4058-bcfa-1a44a56365eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv"
Nov 02 02:04:10.663374 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:10.634973    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj5h9\" (UniqueName: \"kubernetes.io/projected/fc421d13-d643-4058-bcfa-1a44a56365eb-kube-api-access-xj5h9\") pod \"maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv\" (UID: \"fc421d13-d643-4058-bcfa-1a44a56365eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv"
Nov 02 02:04:11.058034 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:11.058001    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc421d13-d643-4058-bcfa-1a44a56365eb-maas-api-tls\") pod \"maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv\" (UID: \"fc421d13-d643-4058-bcfa-1a44a56365eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv"
Nov 02 02:04:11.060979 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:11.060950    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc421d13-d643-4058-bcfa-1a44a56365eb-maas-api-tls\") pod \"maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv\" (UID: \"fc421d13-d643-4058-bcfa-1a44a56365eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv"
Nov 02 02:04:11.325296 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:11.324482    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv"
Nov 02 02:04:11.350580 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:04:11.350545    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:04:11.741038 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:11.741007    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv"]
Nov 02 02:04:11.746563 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:04:11.745708    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc421d13_d643_4058_bcfa_1a44a56365eb.slice/crio-418ab2c33dd35907e4511e9b5ef9bdce910cccb59053d584feaa2a501e58b603 WatchSource:0}: Error finding container 418ab2c33dd35907e4511e9b5ef9bdce910cccb59053d584feaa2a501e58b603: Status 404 returned error can't find the container with id 418ab2c33dd35907e4511e9b5ef9bdce910cccb59053d584feaa2a501e58b603
Nov 02 02:04:11.747823 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:11.747798    5875 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 02:04:12.583872 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:12.583395    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv" event={"ID":"fc421d13-d643-4058-bcfa-1a44a56365eb","Type":"ContainerStarted","Data":"418ab2c33dd35907e4511e9b5ef9bdce910cccb59053d584feaa2a501e58b603"}
Nov 02 02:04:13.520475 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:13.520440    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck"
Nov 02 02:04:13.646885 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:13.646832    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv" event={"ID":"fc421d13-d643-4058-bcfa-1a44a56365eb","Type":"ContainerStarted","Data":"eaf0c26077c94fecc149ebb30cf7d4dcb5a6aa075b59e97ec80bc7af6e9e9161"}
Nov 02 02:04:13.648642 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:13.648051    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv"
Nov 02 02:04:13.770848 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:13.770712    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv" podStartSLOduration=2.871691398 podStartE2EDuration="3.770696996s" totalImagesPullingTime="899.005598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:04:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:04:13.768499272 +0000 UTC m=+1901.209535771" watchObservedRunningTime="2026-11-02 02:04:13.770696996 +0000 UTC m=+1901.211733492"
Nov 02 02:04:17.543402 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:17.543351    5875 patch_prober.go:29] interesting pod/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:04:17.543799 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:17.543427    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx" podUID="160fc571-588a-47fd-9acc-f82010429400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:04:18.765977 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:18.765943    5875 generic.go:356] "Generic (PLEG): container finished" podID="160fc571-588a-47fd-9acc-f82010429400" containerID="7b07b76c1ddf436f96965edd717d17da1e2483fbb0771470eda6cd9f91b68295" exitCode=0
Nov 02 02:04:18.766419 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:18.766006    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx" event={"ID":"160fc571-588a-47fd-9acc-f82010429400","Type":"ContainerDied","Data":"7b07b76c1ddf436f96965edd717d17da1e2483fbb0771470eda6cd9f91b68295"}
Nov 02 02:04:18.766419 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:18.766385    5875 scope.go:122] "RemoveContainer" containerID="7b07b76c1ddf436f96965edd717d17da1e2483fbb0771470eda6cd9f91b68295"
Nov 02 02:04:18.767988 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:18.767960    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:04:18.768059 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:18.768027    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:04:18.768098 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:18.768057    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:04:19.777212 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:19.777176    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx" event={"ID":"160fc571-588a-47fd-9acc-f82010429400","Type":"ContainerStarted","Data":"172403de64910ad436e5c4b34066e68cf28625d4bb802cc3cbeef7e6c0b97a6b"}
Nov 02 02:04:20.283635 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:20.283601    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:04:20.283880 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:20.283785    5875 patch_prober.go:29] interesting pod/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Nov 02 02:04:20.283880 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:20.283841    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx" podUID="160fc571-588a-47fd-9acc-f82010429400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Nov 02 02:04:20.745609 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:20.745581    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv"
Nov 02 02:04:21.283108 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:21.283069    5875 patch_prober.go:29] interesting pod/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Nov 02 02:04:21.283555 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:21.283134    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx" podUID="160fc571-588a-47fd-9acc-f82010429400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Nov 02 02:04:22.283723 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:22.283688    5875 patch_prober.go:29] interesting pod/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Nov 02 02:04:22.284135 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:22.283742    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx" podUID="160fc571-588a-47fd-9acc-f82010429400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Nov 02 02:04:23.283076 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:23.283043    5875 patch_prober.go:29] interesting pod/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Nov 02 02:04:23.283291 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:23.283095    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx" podUID="160fc571-588a-47fd-9acc-f82010429400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Nov 02 02:04:24.283401 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:24.283361    5875 patch_prober.go:29] interesting pod/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Nov 02 02:04:24.283866 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:24.283417    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx" podUID="160fc571-588a-47fd-9acc-f82010429400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Nov 02 02:04:25.287948 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:25.287919    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:04:25.288452 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:25.288432    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:04:25.288976 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:25.288955    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:04:41.328792 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:04:41.328745    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:04:49.627817 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:49.627782    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"]
Nov 02 02:04:49.628457 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:49.628015    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b" podUID="070d0963-3e9f-4e5f-920b-1ae95e0c2e81" containerName="istio-proxy" containerID="cri-o://8ebbdeb439ec450146e441538d4592816c73d72158290fc63944b863c4972775" gracePeriod=30
Nov 02 02:04:49.671944 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:49.671882    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"]
Nov 02 02:04:49.672196 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:49.672143    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r" podUID="7cf2233f-a377-4bee-9eb9-5d5be2a7561c" containerName="istio-proxy" containerID="cri-o://0d70e52871213e4df11c56d740d6f3fd039f73a183d9c54233e354ef461da577" gracePeriod=30
Nov 02 02:04:49.675905 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:49.675877    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"]
Nov 02 02:04:49.676121 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:49.676096    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c" podUID="cf75023b-badb-418a-a93f-48c3d7b0e562" containerName="istio-proxy" containerID="cri-o://c32111e903554f1735e50447faead4ea5b83daedda360f57b77588d16d21456c" gracePeriod=30
Nov 02 02:04:54.905031 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:04:54.904954    5875 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cf2233f_a377_4bee_9eb9_5d5be2a7561c.slice/crio-0d70e52871213e4df11c56d740d6f3fd039f73a183d9c54233e354ef461da577.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cf2233f_a377_4bee_9eb9_5d5be2a7561c.slice/crio-conmon-0d70e52871213e4df11c56d740d6f3fd039f73a183d9c54233e354ef461da577.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 02:04:55.047426 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.047398    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 02:04:55.051393 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.051370    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 02:04:55.074000 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.073982    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 02:04:55.162025 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.161943    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-envoy\") pod \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") "
Nov 02 02:04:55.162025 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.162008    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-credential-socket\") pod \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") "
Nov 02 02:04:55.162266 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.162044    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-token\" (UniqueName: \"kubernetes.io/projected/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-token\") pod \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") "
Nov 02 02:04:55.162266 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.162094    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-podinfo\") pod \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") "
Nov 02 02:04:55.162266 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.162166    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-token\" (UniqueName: \"kubernetes.io/projected/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-token\") pod \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") "
Nov 02 02:04:55.162266 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.162193    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-envoy\") pod \"cf75023b-badb-418a-a93f-48c3d7b0e562\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") "
Nov 02 02:04:55.162266 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.162241    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-data\") pod \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") "
Nov 02 02:04:55.162620 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.162358    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-credential-socket" pod "070d0963-3e9f-4e5f-920b-1ae95e0c2e81" (UID: "070d0963-3e9f-4e5f-920b-1ae95e0c2e81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:55.162620 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.162433    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istiod-ca-cert\") pod \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") "
Nov 02 02:04:55.162620 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.162466    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-data" pod "7cf2233f-a377-4bee-9eb9-5d5be2a7561c" (UID: "7cf2233f-a377-4bee-9eb9-5d5be2a7561c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:55.162620 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.162502    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-workload-certs\") pod \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") "
Nov 02 02:04:55.162834 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.162545    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-credential-socket\") pod \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") "
Nov 02 02:04:55.162834 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.162671    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-token\" (UniqueName: \"kubernetes.io/projected/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-token\") pod \"cf75023b-badb-418a-a93f-48c3d7b0e562\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") "
Nov 02 02:04:55.162834 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.162761    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-podinfo\") pod \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") "
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.163012    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-workload-certs" pod "7cf2233f-a377-4bee-9eb9-5d5be2a7561c" (UID: "7cf2233f-a377-4bee-9eb9-5d5be2a7561c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.163075    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf75023b-badb-418a-a93f-48c3d7b0e562-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf75023b-badb-418a-a93f-48c3d7b0e562-istiod-ca-cert\") pod \"cf75023b-badb-418a-a93f-48c3d7b0e562\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") "
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.163131    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istiod-ca-cert\") pod \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") "
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.163161    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-data\") pod \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") "
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.163187    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-envoy\") pod \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") "
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.163236    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-workload-certs\") pod \"cf75023b-badb-418a-a93f-48c3d7b0e562\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") "
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.163308    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-credential-socket" pod "7cf2233f-a377-4bee-9eb9-5d5be2a7561c" (UID: "7cf2233f-a377-4bee-9eb9-5d5be2a7561c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.163315    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-workload-certs\") pod \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") "
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.163373    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-kube-api-access-nbpkv\" (UniqueName: \"kubernetes.io/projected/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-kube-api-access-nbpkv\") pod \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") "
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.163409    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf75023b-badb-418a-a93f-48c3d7b0e562-kube-api-access-zn9hz\" (UniqueName: \"kubernetes.io/projected/cf75023b-badb-418a-a93f-48c3d7b0e562-kube-api-access-zn9hz\") pod \"cf75023b-badb-418a-a93f-48c3d7b0e562\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") "
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.163482    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-kube-api-access-jpmd9\" (UniqueName: \"kubernetes.io/projected/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-kube-api-access-jpmd9\") pod \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") "
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.163494    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-workload-certs" pod "070d0963-3e9f-4e5f-920b-1ae95e0c2e81" (UID: "070d0963-3e9f-4e5f-920b-1ae95e0c2e81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.163514    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-data\") pod \"cf75023b-badb-418a-a93f-48c3d7b0e562\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") "
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.163543    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-workload-socket\") pod \"cf75023b-badb-418a-a93f-48c3d7b0e562\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") "
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.163680    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-credential-socket\") pod \"cf75023b-badb-418a-a93f-48c3d7b0e562\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") "
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.163747    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-workload-socket\") pod \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\" (UID: \"070d0963-3e9f-4e5f-920b-1ae95e0c2e81\") "
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.163781    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-podinfo\") pod \"cf75023b-badb-418a-a93f-48c3d7b0e562\" (UID: \"cf75023b-badb-418a-a93f-48c3d7b0e562\") "
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.163836    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-workload-socket\") pod \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\" (UID: \"7cf2233f-a377-4bee-9eb9-5d5be2a7561c\") "
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.164196    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istiod-ca-cert" pod "070d0963-3e9f-4e5f-920b-1ae95e0c2e81" (UID: "070d0963-3e9f-4e5f-920b-1ae95e0c2e81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.164570    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-data" pod "cf75023b-badb-418a-a93f-48c3d7b0e562" (UID: "cf75023b-badb-418a-a93f-48c3d7b0e562"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.164912    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.164927    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.164936    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.164945    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.164958    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.164975    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.164994    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.165362    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-workload-socket" pod "7cf2233f-a377-4bee-9eb9-5d5be2a7561c" (UID: "7cf2233f-a377-4bee-9eb9-5d5be2a7561c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.165691    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-envoy" pod "cf75023b-badb-418a-a93f-48c3d7b0e562" (UID: "cf75023b-badb-418a-a93f-48c3d7b0e562"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.165706    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-workload-socket" pod "cf75023b-badb-418a-a93f-48c3d7b0e562" (UID: "cf75023b-badb-418a-a93f-48c3d7b0e562"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.165815    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-token" pod "7cf2233f-a377-4bee-9eb9-5d5be2a7561c" (UID: "7cf2233f-a377-4bee-9eb9-5d5be2a7561c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:55.165882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.165904    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-envoy" pod "7cf2233f-a377-4bee-9eb9-5d5be2a7561c" (UID: "7cf2233f-a377-4bee-9eb9-5d5be2a7561c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:55.167505 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.166380    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-data" pod "070d0963-3e9f-4e5f-920b-1ae95e0c2e81" (UID: "070d0963-3e9f-4e5f-920b-1ae95e0c2e81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:55.167602 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.167572    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf75023b-badb-418a-a93f-48c3d7b0e562-istiod-ca-cert" pod "cf75023b-badb-418a-a93f-48c3d7b0e562" (UID: "cf75023b-badb-418a-a93f-48c3d7b0e562"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:04:55.167602 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.167581    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-credential-socket" pod "cf75023b-badb-418a-a93f-48c3d7b0e562" (UID: "cf75023b-badb-418a-a93f-48c3d7b0e562"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:55.167786 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.167767    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-workload-socket" pod "070d0963-3e9f-4e5f-920b-1ae95e0c2e81" (UID: "070d0963-3e9f-4e5f-920b-1ae95e0c2e81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:55.167855 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.167842    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-workload-certs" pod "cf75023b-badb-418a-a93f-48c3d7b0e562" (UID: "cf75023b-badb-418a-a93f-48c3d7b0e562"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:55.168343 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.168298    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istiod-ca-cert" pod "7cf2233f-a377-4bee-9eb9-5d5be2a7561c" (UID: "7cf2233f-a377-4bee-9eb9-5d5be2a7561c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:04:55.168840 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.168527    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-envoy" pod "070d0963-3e9f-4e5f-920b-1ae95e0c2e81" (UID: "070d0963-3e9f-4e5f-920b-1ae95e0c2e81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:55.168840 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.168558    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf75023b-badb-418a-a93f-48c3d7b0e562-kube-api-access-zn9hz" pod "cf75023b-badb-418a-a93f-48c3d7b0e562" (UID: "cf75023b-badb-418a-a93f-48c3d7b0e562"). InnerVolumeSpecName "kube-api-access-zn9hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:55.168840 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.168624    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-token" pod "cf75023b-badb-418a-a93f-48c3d7b0e562" (UID: "cf75023b-badb-418a-a93f-48c3d7b0e562"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:55.168840 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.168665    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-podinfo" pod "070d0963-3e9f-4e5f-920b-1ae95e0c2e81" (UID: "070d0963-3e9f-4e5f-920b-1ae95e0c2e81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:04:55.168840 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.168672    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-podinfo" pod "7cf2233f-a377-4bee-9eb9-5d5be2a7561c" (UID: "7cf2233f-a377-4bee-9eb9-5d5be2a7561c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:04:55.169176 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.169056    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-token" pod "070d0963-3e9f-4e5f-920b-1ae95e0c2e81" (UID: "070d0963-3e9f-4e5f-920b-1ae95e0c2e81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:55.170014 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.169987    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-podinfo" pod "cf75023b-badb-418a-a93f-48c3d7b0e562" (UID: "cf75023b-badb-418a-a93f-48c3d7b0e562"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:04:55.170119 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.170098    5875 generic.go:356] "Generic (PLEG): container finished" podID="7cf2233f-a377-4bee-9eb9-5d5be2a7561c" containerID="0d70e52871213e4df11c56d740d6f3fd039f73a183d9c54233e354ef461da577" exitCode=0
Nov 02 02:04:55.170166 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.170153    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"
Nov 02 02:04:55.170264 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.170189    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r" event={"ID":"7cf2233f-a377-4bee-9eb9-5d5be2a7561c","Type":"ContainerDied","Data":"0d70e52871213e4df11c56d740d6f3fd039f73a183d9c54233e354ef461da577"}
Nov 02 02:04:55.170336 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.170301    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r" event={"ID":"7cf2233f-a377-4bee-9eb9-5d5be2a7561c","Type":"ContainerDied","Data":"4bd52cd57449dfe3436680158063cbc1b3c015d61c6a4c288c2942c53de19f05"}
Nov 02 02:04:55.170394 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.170337    5875 scope.go:122] "RemoveContainer" containerID="0d70e52871213e4df11c56d740d6f3fd039f73a183d9c54233e354ef461da577"
Nov 02 02:04:55.170862 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.170839    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-kube-api-access-nbpkv" pod "7cf2233f-a377-4bee-9eb9-5d5be2a7561c" (UID: "7cf2233f-a377-4bee-9eb9-5d5be2a7561c"). InnerVolumeSpecName "kube-api-access-nbpkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:55.171043 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.171022    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-kube-api-access-jpmd9" pod "070d0963-3e9f-4e5f-920b-1ae95e0c2e81" (UID: "070d0963-3e9f-4e5f-920b-1ae95e0c2e81"). InnerVolumeSpecName "kube-api-access-jpmd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:55.172207 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.172181    5875 generic.go:356] "Generic (PLEG): container finished" podID="070d0963-3e9f-4e5f-920b-1ae95e0c2e81" containerID="8ebbdeb439ec450146e441538d4592816c73d72158290fc63944b863c4972775" exitCode=0
Nov 02 02:04:55.172340 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.172247    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"
Nov 02 02:04:55.172340 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.172326    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b" event={"ID":"070d0963-3e9f-4e5f-920b-1ae95e0c2e81","Type":"ContainerDied","Data":"8ebbdeb439ec450146e441538d4592816c73d72158290fc63944b863c4972775"}
Nov 02 02:04:55.172442 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.172371    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b" event={"ID":"070d0963-3e9f-4e5f-920b-1ae95e0c2e81","Type":"ContainerDied","Data":"4cf8fb954577f7c82b22122b858ae7c924485cfe0a54be7ada0de34eb47e2b1e"}
Nov 02 02:04:55.173963 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.173929    5875 generic.go:356] "Generic (PLEG): container finished" podID="cf75023b-badb-418a-a93f-48c3d7b0e562" containerID="c32111e903554f1735e50447faead4ea5b83daedda360f57b77588d16d21456c" exitCode=0
Nov 02 02:04:55.174038 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.173970    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"
Nov 02 02:04:55.174038 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.173985    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c" event={"ID":"cf75023b-badb-418a-a93f-48c3d7b0e562","Type":"ContainerDied","Data":"c32111e903554f1735e50447faead4ea5b83daedda360f57b77588d16d21456c"}
Nov 02 02:04:55.174136 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.174027    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c" event={"ID":"cf75023b-badb-418a-a93f-48c3d7b0e562","Type":"ContainerDied","Data":"5f5742337d6d3eb11285aa25772da1a7a9a2418adfcba0882c6187c52201b398"}
Nov 02 02:04:55.184371 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.184349    5875 scope.go:122] "RemoveContainer" containerID="0d70e52871213e4df11c56d740d6f3fd039f73a183d9c54233e354ef461da577"
Nov 02 02:04:55.185611 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:04:55.185586    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d70e52871213e4df11c56d740d6f3fd039f73a183d9c54233e354ef461da577\": container with ID starting with 0d70e52871213e4df11c56d740d6f3fd039f73a183d9c54233e354ef461da577 not found: ID does not exist" containerID="0d70e52871213e4df11c56d740d6f3fd039f73a183d9c54233e354ef461da577"
Nov 02 02:04:55.185702 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.185622    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d70e52871213e4df11c56d740d6f3fd039f73a183d9c54233e354ef461da577"} err="failed to get container status \"0d70e52871213e4df11c56d740d6f3fd039f73a183d9c54233e354ef461da577\": rpc error: code = NotFound desc = could not find container \"0d70e52871213e4df11c56d740d6f3fd039f73a183d9c54233e354ef461da577\": container with ID starting with 0d70e52871213e4df11c56d740d6f3fd039f73a183d9c54233e354ef461da577 not found: ID does not exist"
Nov 02 02:04:55.185702 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.185647    5875 scope.go:122] "RemoveContainer" containerID="8ebbdeb439ec450146e441538d4592816c73d72158290fc63944b863c4972775"
Nov 02 02:04:55.198445 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.198428    5875 scope.go:122] "RemoveContainer" containerID="8ebbdeb439ec450146e441538d4592816c73d72158290fc63944b863c4972775"
Nov 02 02:04:55.198716 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:04:55.198698    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ebbdeb439ec450146e441538d4592816c73d72158290fc63944b863c4972775\": container with ID starting with 8ebbdeb439ec450146e441538d4592816c73d72158290fc63944b863c4972775 not found: ID does not exist" containerID="8ebbdeb439ec450146e441538d4592816c73d72158290fc63944b863c4972775"
Nov 02 02:04:55.198783 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.198723    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ebbdeb439ec450146e441538d4592816c73d72158290fc63944b863c4972775"} err="failed to get container status \"8ebbdeb439ec450146e441538d4592816c73d72158290fc63944b863c4972775\": rpc error: code = NotFound desc = could not find container \"8ebbdeb439ec450146e441538d4592816c73d72158290fc63944b863c4972775\": container with ID starting with 8ebbdeb439ec450146e441538d4592816c73d72158290fc63944b863c4972775 not found: ID does not exist"
Nov 02 02:04:55.198783 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.198741    5875 scope.go:122] "RemoveContainer" containerID="c32111e903554f1735e50447faead4ea5b83daedda360f57b77588d16d21456c"
Nov 02 02:04:55.208576 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.208561    5875 scope.go:122] "RemoveContainer" containerID="c32111e903554f1735e50447faead4ea5b83daedda360f57b77588d16d21456c"
Nov 02 02:04:55.208846 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:04:55.208827    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c32111e903554f1735e50447faead4ea5b83daedda360f57b77588d16d21456c\": container with ID starting with c32111e903554f1735e50447faead4ea5b83daedda360f57b77588d16d21456c not found: ID does not exist" containerID="c32111e903554f1735e50447faead4ea5b83daedda360f57b77588d16d21456c"
Nov 02 02:04:55.208913 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.208854    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c32111e903554f1735e50447faead4ea5b83daedda360f57b77588d16d21456c"} err="failed to get container status \"c32111e903554f1735e50447faead4ea5b83daedda360f57b77588d16d21456c\": rpc error: code = NotFound desc = could not find container \"c32111e903554f1735e50447faead4ea5b83daedda360f57b77588d16d21456c\": container with ID starting with c32111e903554f1735e50447faead4ea5b83daedda360f57b77588d16d21456c not found: ID does not exist"
Nov 02 02:04:55.267385 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.267346    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.267385 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.267385    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf75023b-badb-418a-a93f-48c3d7b0e562-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.267619 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.267400    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.267619 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.267414    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.267619 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.267428    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.267619 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.267443    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbpkv\" (UniqueName: \"kubernetes.io/projected/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-kube-api-access-nbpkv\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.267619 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.267462    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn9hz\" (UniqueName: \"kubernetes.io/projected/cf75023b-badb-418a-a93f-48c3d7b0e562-kube-api-access-zn9hz\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.267619 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.267477    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpmd9\" (UniqueName: \"kubernetes.io/projected/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-kube-api-access-jpmd9\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.267619 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.267530    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.267619 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.267546    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.267619 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.267562    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.267619 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.267576    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.267619 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.267592    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.267619 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.267605    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.267619 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.267619    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.268351 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.267634    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.268351 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.267647    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070d0963-3e9f-4e5f-920b-1ae95e0c2e81-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.268351 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.267661    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.268351 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.267675    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cf2233f-a377-4bee-9eb9-5d5be2a7561c-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.268351 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.267689    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf75023b-badb-418a-a93f-48c3d7b0e562-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:55.412689 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.412603    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"]
Nov 02 02:04:55.418293 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.418261    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ts98c"]
Nov 02 02:04:55.597362 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.597301    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"]
Nov 02 02:04:55.600923 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.600891    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-kqj2b"]
Nov 02 02:04:55.770090 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.770016    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"]
Nov 02 02:04:55.776254 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:55.776209    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lc78r"]
Nov 02 02:04:56.699113 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:56.699085    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="070d0963-3e9f-4e5f-920b-1ae95e0c2e81" path="/var/lib/kubelet/pods/070d0963-3e9f-4e5f-920b-1ae95e0c2e81/volumes"
Nov 02 02:04:56.699609 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:56.699593    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cf2233f-a377-4bee-9eb9-5d5be2a7561c" path="/var/lib/kubelet/pods/7cf2233f-a377-4bee-9eb9-5d5be2a7561c/volumes"
Nov 02 02:04:56.700157 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:56.700138    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf75023b-badb-418a-a93f-48c3d7b0e562" path="/var/lib/kubelet/pods/cf75023b-badb-418a-a93f-48c3d7b0e562/volumes"
Nov 02 02:04:58.245829 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.245795    5875 generic.go:356] "Generic (PLEG): container finished" podID="eb67a90f-51c6-4440-923d-db0edea22576" containerID="218cfc7752e8ecb3b44a5d1566a3b02e44e0596529b3793861d7ecf021601e32" exitCode=0
Nov 02 02:04:58.246387 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.245875    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592" event={"ID":"eb67a90f-51c6-4440-923d-db0edea22576","Type":"ContainerDied","Data":"218cfc7752e8ecb3b44a5d1566a3b02e44e0596529b3793861d7ecf021601e32"}
Nov 02 02:04:58.246461 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.246423    5875 scope.go:122] "RemoveContainer" containerID="218cfc7752e8ecb3b44a5d1566a3b02e44e0596529b3793861d7ecf021601e32"
Nov 02 02:04:58.248744 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.248705    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:04:58.248887 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.248801    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:04:58.248887 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.248854    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:04:58.592984 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.592950    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm"]
Nov 02 02:04:58.593796 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.593780    5875 cpu_manager.go:414] "containerMap: removing container" podUID="070d0963-3e9f-4e5f-920b-1ae95e0c2e81" containerName="istio-proxy"
Nov 02 02:04:58.593861 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.593797    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="070d0963-3e9f-4e5f-920b-1ae95e0c2e81" containerName="istio-proxy"
Nov 02 02:04:58.593861 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.593814    5875 cpu_manager.go:414] "containerMap: removing container" podUID="cf75023b-badb-418a-a93f-48c3d7b0e562" containerName="istio-proxy"
Nov 02 02:04:58.593861 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.593819    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf75023b-badb-418a-a93f-48c3d7b0e562" containerName="istio-proxy"
Nov 02 02:04:58.593861 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.593835    5875 cpu_manager.go:414] "containerMap: removing container" podUID="7cf2233f-a377-4bee-9eb9-5d5be2a7561c" containerName="istio-proxy"
Nov 02 02:04:58.593861 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.593840    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cf2233f-a377-4bee-9eb9-5d5be2a7561c" containerName="istio-proxy"
Nov 02 02:04:58.594021 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.593958    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="070d0963-3e9f-4e5f-920b-1ae95e0c2e81" containerName="istio-proxy"
Nov 02 02:04:58.594021 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.593969    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf75023b-badb-418a-a93f-48c3d7b0e562" containerName="istio-proxy"
Nov 02 02:04:58.594021 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.593978    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cf2233f-a377-4bee-9eb9-5d5be2a7561c" containerName="istio-proxy"
Nov 02 02:04:58.597771 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.597754    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm"
Nov 02 02:04:58.651670 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.651522    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7a24b9-serving-cert\""
Nov 02 02:04:58.656367 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.656320    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm"]
Nov 02 02:04:58.751681 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.751648    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b93b73a-53d8-463f-8f68-4490e7a8253f-maas-api-tls\") pod \"maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm\" (UID: \"6b93b73a-53d8-463f-8f68-4490e7a8253f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm"
Nov 02 02:04:58.751970 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.751934    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b93b73a-53d8-463f-8f68-4490e7a8253f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm\" (UID: \"6b93b73a-53d8-463f-8f68-4490e7a8253f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm"
Nov 02 02:04:58.752166 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.752148    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b93b73a-53d8-463f-8f68-4490e7a8253f-metrics-certs\") pod \"maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm\" (UID: \"6b93b73a-53d8-463f-8f68-4490e7a8253f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm"
Nov 02 02:04:58.752290 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.752268    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv4lw\" (UniqueName: \"kubernetes.io/projected/6b93b73a-53d8-463f-8f68-4490e7a8253f-kube-api-access-zv4lw\") pod \"maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm\" (UID: \"6b93b73a-53d8-463f-8f68-4490e7a8253f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm"
Nov 02 02:04:58.858430 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.858307    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b93b73a-53d8-463f-8f68-4490e7a8253f-maas-api-tls\") pod \"maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm\" (UID: \"6b93b73a-53d8-463f-8f68-4490e7a8253f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm"
Nov 02 02:04:58.858430 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.858410    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b93b73a-53d8-463f-8f68-4490e7a8253f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm\" (UID: \"6b93b73a-53d8-463f-8f68-4490e7a8253f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm"
Nov 02 02:04:58.858689 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.858589    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b93b73a-53d8-463f-8f68-4490e7a8253f-metrics-certs\") pod \"maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm\" (UID: \"6b93b73a-53d8-463f-8f68-4490e7a8253f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm"
Nov 02 02:04:58.858758 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.858687    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv4lw\" (UniqueName: \"kubernetes.io/projected/6b93b73a-53d8-463f-8f68-4490e7a8253f-kube-api-access-zv4lw\") pod \"maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm\" (UID: \"6b93b73a-53d8-463f-8f68-4490e7a8253f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm"
Nov 02 02:04:58.859550 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.859507    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b93b73a-53d8-463f-8f68-4490e7a8253f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm\" (UID: \"6b93b73a-53d8-463f-8f68-4490e7a8253f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm"
Nov 02 02:04:58.861481 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.861458    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b93b73a-53d8-463f-8f68-4490e7a8253f-maas-api-tls\") pod \"maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm\" (UID: \"6b93b73a-53d8-463f-8f68-4490e7a8253f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm"
Nov 02 02:04:58.861872 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.861851    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b93b73a-53d8-463f-8f68-4490e7a8253f-metrics-certs\") pod \"maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm\" (UID: \"6b93b73a-53d8-463f-8f68-4490e7a8253f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm"
Nov 02 02:04:58.924469 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:58.924430    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv4lw\" (UniqueName: \"kubernetes.io/projected/6b93b73a-53d8-463f-8f68-4490e7a8253f-kube-api-access-zv4lw\") pod \"maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm\" (UID: \"6b93b73a-53d8-463f-8f68-4490e7a8253f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm"
Nov 02 02:04:59.217246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:59.215515    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm"
Nov 02 02:04:59.257978 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:59.257945    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592" event={"ID":"eb67a90f-51c6-4440-923d-db0edea22576","Type":"ContainerStarted","Data":"fda56a66cfddb674c9b940aac1ffd002b969a38238262737bbd37383a1ee7b48"}
Nov 02 02:04:59.629427 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:04:59.629396    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b93b73a_53d8_463f_8f68_4490e7a8253f.slice/crio-8412853f880f94bce7e87dd0ca627762fb9d3650b79daa1c1e776c50a901b7b3 WatchSource:0}: Error finding container 8412853f880f94bce7e87dd0ca627762fb9d3650b79daa1c1e776c50a901b7b3: Status 404 returned error can't find the container with id 8412853f880f94bce7e87dd0ca627762fb9d3650b79daa1c1e776c50a901b7b3
Nov 02 02:04:59.629885 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:59.629863    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm"]
Nov 02 02:04:59.748398 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:59.748367    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:04:59.749531 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:59.749502    5875 patch_prober.go:29] interesting pod/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 02 02:04:59.749676 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:04:59.749566    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592" podUID="eb67a90f-51c6-4440-923d-db0edea22576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 02 02:05:00.147568 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:00.147534    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:05:00.267053 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:00.267017    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm" event={"ID":"6b93b73a-53d8-463f-8f68-4490e7a8253f","Type":"ContainerStarted","Data":"8412853f880f94bce7e87dd0ca627762fb9d3650b79daa1c1e776c50a901b7b3"}
Nov 02 02:05:00.749047 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:00.749012    5875 patch_prober.go:29] interesting pod/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 02 02:05:00.749276 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:00.749085    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592" podUID="eb67a90f-51c6-4440-923d-db0edea22576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 02 02:05:01.276991 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:01.276955    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm" event={"ID":"6b93b73a-53d8-463f-8f68-4490e7a8253f","Type":"ContainerStarted","Data":"fee85df8658ace5e0511e159c941ac7a92f9ad86fb52c40326e4dd6abb3a1b91"}
Nov 02 02:05:01.277514 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:01.277053    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm"
Nov 02 02:05:01.426457 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:01.426389    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm" podStartSLOduration=2.491116991 podStartE2EDuration="3.426368581s" totalImagesPullingTime="935.25159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:05:01.396024125 +0000 UTC m=+1948.837060619" watchObservedRunningTime="2026-11-02 02:05:01.426368581 +0000 UTC m=+1948.867405080"
Nov 02 02:05:01.748815 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:01.748781    5875 patch_prober.go:29] interesting pod/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 02 02:05:01.748979 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:01.748843    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592" podUID="eb67a90f-51c6-4440-923d-db0edea22576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 02 02:05:02.749105 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:02.749070    5875 patch_prober.go:29] interesting pod/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 02 02:05:02.749549 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:02.749132    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592" podUID="eb67a90f-51c6-4440-923d-db0edea22576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 02 02:05:03.749459 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:03.749428    5875 patch_prober.go:29] interesting pod/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 02 02:05:03.749868 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:03.749479    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592" podUID="eb67a90f-51c6-4440-923d-db0edea22576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 02 02:05:03.826759 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:03.826716    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm"]
Nov 02 02:05:03.831632 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:03.831611    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm"
Nov 02 02:05:03.886403 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:03.886371    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm"]
Nov 02 02:05:03.928719 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:03.928678    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-d1405a-serving-cert\""
Nov 02 02:05:03.973145 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:03.973105    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm\" (UID: \"2572e86e-fe1c-4b20-ba3f-9eb3270b6127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm"
Nov 02 02:05:03.973449 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:03.973427    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm\" (UID: \"2572e86e-fe1c-4b20-ba3f-9eb3270b6127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm"
Nov 02 02:05:03.973610 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:03.973596    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9kx6\" (UniqueName: \"kubernetes.io/projected/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-kube-api-access-k9kx6\") pod \"maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm\" (UID: \"2572e86e-fe1c-4b20-ba3f-9eb3270b6127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm"
Nov 02 02:05:03.973756 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:03.973738    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm\" (UID: \"2572e86e-fe1c-4b20-ba3f-9eb3270b6127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm"
Nov 02 02:05:04.116297 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:04.082080    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm\" (UID: \"2572e86e-fe1c-4b20-ba3f-9eb3270b6127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm"
Nov 02 02:05:04.116297 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:04.094688    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm\" (UID: \"2572e86e-fe1c-4b20-ba3f-9eb3270b6127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm"
Nov 02 02:05:04.116297 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:04.094781    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9kx6\" (UniqueName: \"kubernetes.io/projected/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-kube-api-access-k9kx6\") pod \"maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm\" (UID: \"2572e86e-fe1c-4b20-ba3f-9eb3270b6127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm"
Nov 02 02:05:04.116297 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:04.094847    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm\" (UID: \"2572e86e-fe1c-4b20-ba3f-9eb3270b6127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm"
Nov 02 02:05:04.116297 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:04.096068    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm\" (UID: \"2572e86e-fe1c-4b20-ba3f-9eb3270b6127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm"
Nov 02 02:05:04.116297 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:05:04.091415    5875 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-serving-cert: secret "maas-api-e2e-shared-b-w2-d1405a-serving-cert" not found
Nov 02 02:05:04.116297 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:05:04.096173    5875 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-maas-api-tls podName:2572e86e-fe1c-4b20-ba3f-9eb3270b6127 nodeName:}" failed. No retries permitted until 2026-11-02 02:05:04.596150303 +0000 UTC m=+1952.037186778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-maas-api-tls") pod "maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm" (UID: "2572e86e-fe1c-4b20-ba3f-9eb3270b6127") : secret "maas-api-e2e-shared-b-w2-d1405a-serving-cert" not found
Nov 02 02:05:04.134256 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:04.119367    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm\" (UID: \"2572e86e-fe1c-4b20-ba3f-9eb3270b6127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm"
Nov 02 02:05:04.159898 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:04.159868    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9kx6\" (UniqueName: \"kubernetes.io/projected/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-kube-api-access-k9kx6\") pod \"maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm\" (UID: \"2572e86e-fe1c-4b20-ba3f-9eb3270b6127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm"
Nov 02 02:05:04.618290 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:04.618254    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm\" (UID: \"2572e86e-fe1c-4b20-ba3f-9eb3270b6127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm"
Nov 02 02:05:04.620827 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:04.620803    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm\" (UID: \"2572e86e-fe1c-4b20-ba3f-9eb3270b6127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm"
Nov 02 02:05:04.746015 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:04.745988    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm"
Nov 02 02:05:04.754610 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:04.754588    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:05:04.755833 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:04.755812    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:05:05.128652 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:05.128622    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm"]
Nov 02 02:05:05.129837 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:05:05.129809    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2572e86e_fe1c_4b20_ba3f_9eb3270b6127.slice/crio-bd26469a0b187c9b0c4ba899f689560a4ab2d24c19dae329f71c5e19e7941af9 WatchSource:0}: Error finding container bd26469a0b187c9b0c4ba899f689560a4ab2d24c19dae329f71c5e19e7941af9: Status 404 returned error can't find the container with id bd26469a0b187c9b0c4ba899f689560a4ab2d24c19dae329f71c5e19e7941af9
Nov 02 02:05:05.343489 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:05.343455    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm" event={"ID":"2572e86e-fe1c-4b20-ba3f-9eb3270b6127","Type":"ContainerStarted","Data":"bd26469a0b187c9b0c4ba899f689560a4ab2d24c19dae329f71c5e19e7941af9"}
Nov 02 02:05:06.352945 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:06.352862    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm" event={"ID":"2572e86e-fe1c-4b20-ba3f-9eb3270b6127","Type":"ContainerStarted","Data":"bfd1599b2d4968997bc3e992a4ce24407aacd87e7ab882192b8644954b818096"}
Nov 02 02:05:06.352945 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:06.352926    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm"
Nov 02 02:05:06.472043 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:06.471991    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm" podStartSLOduration=2.637776822 podStartE2EDuration="3.471974551s" totalImagesPullingTime="834.197729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:05:06.468628129 +0000 UTC m=+1953.909664651" watchObservedRunningTime="2026-11-02 02:05:06.471974551 +0000 UTC m=+1953.913011047"
Nov 02 02:05:07.290446 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:07.290420    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm"
Nov 02 02:05:11.323929 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:05:11.323886    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:05:12.366970 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:12.366938    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm"
Nov 02 02:05:13.479231 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:13.479183    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:05:14.925859 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:14.925820    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv"]
Nov 02 02:05:15.037943 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:15.037909    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8"]
Nov 02 02:05:19.926838 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:19.926792    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv" podUID="40416082-9ca7-4bb0-91f8-534092c7d0c1" containerName="payload-processing" containerID="cri-o://1f43212117a3f8c09ef419b1cddede7da589e1b1b02d976e3102e91b089e79c0" gracePeriod=25
Nov 02 02:05:20.038509 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.038446    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8" podUID="bb3138b1-f0f9-4801-9ba7-3ed676116fca" containerName="payload-pre-processing" containerID="cri-o://e377507562999a6578c4afaca92c71cf0358bf59f086740b5807c6b935e544c0" gracePeriod=25
Nov 02 02:05:20.291387 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.291212    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv"
Nov 02 02:05:20.384972 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.384944    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40416082-9ca7-4bb0-91f8-534092c7d0c1-kube-api-access-vxbtl\" (UniqueName: \"kubernetes.io/projected/40416082-9ca7-4bb0-91f8-534092c7d0c1-kube-api-access-vxbtl\") pod \"40416082-9ca7-4bb0-91f8-534092c7d0c1\" (UID: \"40416082-9ca7-4bb0-91f8-534092c7d0c1\") "
Nov 02 02:05:20.385112 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.385014    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40416082-9ca7-4bb0-91f8-534092c7d0c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40416082-9ca7-4bb0-91f8-534092c7d0c1-plugins-config-volume\") pod \"40416082-9ca7-4bb0-91f8-534092c7d0c1\" (UID: \"40416082-9ca7-4bb0-91f8-534092c7d0c1\") "
Nov 02 02:05:20.385358 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.385335    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40416082-9ca7-4bb0-91f8-534092c7d0c1-plugins-config-volume" pod "40416082-9ca7-4bb0-91f8-534092c7d0c1" (UID: "40416082-9ca7-4bb0-91f8-534092c7d0c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:05:20.385815 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.385477    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40416082-9ca7-4bb0-91f8-534092c7d0c1-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:20.387393 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.387370    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40416082-9ca7-4bb0-91f8-534092c7d0c1-kube-api-access-vxbtl" pod "40416082-9ca7-4bb0-91f8-534092c7d0c1" (UID: "40416082-9ca7-4bb0-91f8-534092c7d0c1"). InnerVolumeSpecName "kube-api-access-vxbtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:05:20.428966 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.428659    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8"
Nov 02 02:05:20.485959 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.485929    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb3138b1-f0f9-4801-9ba7-3ed676116fca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb3138b1-f0f9-4801-9ba7-3ed676116fca-plugins-config-volume\") pod \"bb3138b1-f0f9-4801-9ba7-3ed676116fca\" (UID: \"bb3138b1-f0f9-4801-9ba7-3ed676116fca\") "
Nov 02 02:05:20.485959 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.485963    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3138b1-f0f9-4801-9ba7-3ed676116fca-kube-api-access-6b5rh\" (UniqueName: \"kubernetes.io/projected/bb3138b1-f0f9-4801-9ba7-3ed676116fca-kube-api-access-6b5rh\") pod \"bb3138b1-f0f9-4801-9ba7-3ed676116fca\" (UID: \"bb3138b1-f0f9-4801-9ba7-3ed676116fca\") "
Nov 02 02:05:20.486386 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.486359    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3138b1-f0f9-4801-9ba7-3ed676116fca-plugins-config-volume" pod "bb3138b1-f0f9-4801-9ba7-3ed676116fca" (UID: "bb3138b1-f0f9-4801-9ba7-3ed676116fca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:05:20.486698 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.486680    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb3138b1-f0f9-4801-9ba7-3ed676116fca-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:20.486768 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.486704    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxbtl\" (UniqueName: \"kubernetes.io/projected/40416082-9ca7-4bb0-91f8-534092c7d0c1-kube-api-access-vxbtl\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:20.488214 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.488193    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3138b1-f0f9-4801-9ba7-3ed676116fca-kube-api-access-6b5rh" pod "bb3138b1-f0f9-4801-9ba7-3ed676116fca" (UID: "bb3138b1-f0f9-4801-9ba7-3ed676116fca"). InnerVolumeSpecName "kube-api-access-6b5rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:05:20.575239 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.575118    5875 generic.go:356] "Generic (PLEG): container finished" podID="40416082-9ca7-4bb0-91f8-534092c7d0c1" containerID="1f43212117a3f8c09ef419b1cddede7da589e1b1b02d976e3102e91b089e79c0" exitCode=0
Nov 02 02:05:20.575239 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.575162    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv"
Nov 02 02:05:20.575493 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.575216    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv" event={"ID":"40416082-9ca7-4bb0-91f8-534092c7d0c1","Type":"ContainerDied","Data":"1f43212117a3f8c09ef419b1cddede7da589e1b1b02d976e3102e91b089e79c0"}
Nov 02 02:05:20.575493 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.575341    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv" event={"ID":"40416082-9ca7-4bb0-91f8-534092c7d0c1","Type":"ContainerDied","Data":"8e1ea399e52b1bebec0cda1feb7d52614be61a6efcd4e7309eb48d27c2c2fa4a"}
Nov 02 02:05:20.575493 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.575371    5875 scope.go:122] "RemoveContainer" containerID="1f43212117a3f8c09ef419b1cddede7da589e1b1b02d976e3102e91b089e79c0"
Nov 02 02:05:20.577133 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.577094    5875 generic.go:356] "Generic (PLEG): container finished" podID="bb3138b1-f0f9-4801-9ba7-3ed676116fca" containerID="e377507562999a6578c4afaca92c71cf0358bf59f086740b5807c6b935e544c0" exitCode=0
Nov 02 02:05:20.577305 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.577154    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8"
Nov 02 02:05:20.577305 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.577269    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8" event={"ID":"bb3138b1-f0f9-4801-9ba7-3ed676116fca","Type":"ContainerDied","Data":"e377507562999a6578c4afaca92c71cf0358bf59f086740b5807c6b935e544c0"}
Nov 02 02:05:20.577893 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.577341    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8" event={"ID":"bb3138b1-f0f9-4801-9ba7-3ed676116fca","Type":"ContainerDied","Data":"9b91f7111bbfe225f224141271c605cdaa2d6e1db5abe6ba382744ba86ad8dd3"}
Nov 02 02:05:20.588123 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.588097    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b5rh\" (UniqueName: \"kubernetes.io/projected/bb3138b1-f0f9-4801-9ba7-3ed676116fca-kube-api-access-6b5rh\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:20.594419 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.594353    5875 scope.go:122] "RemoveContainer" containerID="1f43212117a3f8c09ef419b1cddede7da589e1b1b02d976e3102e91b089e79c0"
Nov 02 02:05:20.597064 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:05:20.597039    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f43212117a3f8c09ef419b1cddede7da589e1b1b02d976e3102e91b089e79c0\": container with ID starting with 1f43212117a3f8c09ef419b1cddede7da589e1b1b02d976e3102e91b089e79c0 not found: ID does not exist" containerID="1f43212117a3f8c09ef419b1cddede7da589e1b1b02d976e3102e91b089e79c0"
Nov 02 02:05:20.602271 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.597195    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f43212117a3f8c09ef419b1cddede7da589e1b1b02d976e3102e91b089e79c0"} err="failed to get container status \"1f43212117a3f8c09ef419b1cddede7da589e1b1b02d976e3102e91b089e79c0\": rpc error: code = NotFound desc = could not find container \"1f43212117a3f8c09ef419b1cddede7da589e1b1b02d976e3102e91b089e79c0\": container with ID starting with 1f43212117a3f8c09ef419b1cddede7da589e1b1b02d976e3102e91b089e79c0 not found: ID does not exist"
Nov 02 02:05:20.602407 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.602388    5875 scope.go:122] "RemoveContainer" containerID="e377507562999a6578c4afaca92c71cf0358bf59f086740b5807c6b935e544c0"
Nov 02 02:05:20.612889 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.612869    5875 scope.go:122] "RemoveContainer" containerID="e377507562999a6578c4afaca92c71cf0358bf59f086740b5807c6b935e544c0"
Nov 02 02:05:20.613201 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:05:20.613181    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e377507562999a6578c4afaca92c71cf0358bf59f086740b5807c6b935e544c0\": container with ID starting with e377507562999a6578c4afaca92c71cf0358bf59f086740b5807c6b935e544c0 not found: ID does not exist" containerID="e377507562999a6578c4afaca92c71cf0358bf59f086740b5807c6b935e544c0"
Nov 02 02:05:20.613284 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.613210    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e377507562999a6578c4afaca92c71cf0358bf59f086740b5807c6b935e544c0"} err="failed to get container status \"e377507562999a6578c4afaca92c71cf0358bf59f086740b5807c6b935e544c0\": rpc error: code = NotFound desc = could not find container \"e377507562999a6578c4afaca92c71cf0358bf59f086740b5807c6b935e544c0\": container with ID starting with e377507562999a6578c4afaca92c71cf0358bf59f086740b5807c6b935e544c0 not found: ID does not exist"
Nov 02 02:05:20.740353 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.740318    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv"]
Nov 02 02:05:20.749206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.749178    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bd3971-f9c678567-8rkjv"]
Nov 02 02:05:20.867164 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.867056    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8"]
Nov 02 02:05:20.870679 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:20.870656    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bd3971-b455b6b9f-jsgd8"]
Nov 02 02:05:22.700283 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:22.700250    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40416082-9ca7-4bb0-91f8-534092c7d0c1" path="/var/lib/kubelet/pods/40416082-9ca7-4bb0-91f8-534092c7d0c1/volumes"
Nov 02 02:05:22.700743 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:22.700704    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb3138b1-f0f9-4801-9ba7-3ed676116fca" path="/var/lib/kubelet/pods/bb3138b1-f0f9-4801-9ba7-3ed676116fca/volumes"
Nov 02 02:05:24.479165 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:24.477489    5875 patch_prober.go:29] interesting pod/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": read tcp 10.134.0.2:46744->10.134.0.158:15021: read: connection reset by peer" start-of-body=
Nov 02 02:05:24.479165 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:24.477548    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb" podUID="0931244f-232a-4588-8b26-eb3b5b5773c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": read tcp 10.134.0.2:46744->10.134.0.158:15021: read: connection reset by peer"
Nov 02 02:05:24.614107 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:24.614071    5875 generic.go:356] "Generic (PLEG): container finished" podID="0931244f-232a-4588-8b26-eb3b5b5773c2" containerID="929b255fdf84f63a74a2f0290af1e9bb6a189e5bb6512e7a1665d741a414e661" exitCode=0
Nov 02 02:05:24.614287 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:24.614150    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb" event={"ID":"0931244f-232a-4588-8b26-eb3b5b5773c2","Type":"ContainerDied","Data":"929b255fdf84f63a74a2f0290af1e9bb6a189e5bb6512e7a1665d741a414e661"}
Nov 02 02:05:24.614620 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:24.614605    5875 scope.go:122] "RemoveContainer" containerID="929b255fdf84f63a74a2f0290af1e9bb6a189e5bb6512e7a1665d741a414e661"
Nov 02 02:05:24.616414 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:24.616385    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:05:24.616525 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:24.616456    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:05:24.616525 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:24.616487    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:05:25.629671 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:25.629637    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb" event={"ID":"0931244f-232a-4588-8b26-eb3b5b5773c2","Type":"ContainerStarted","Data":"dd6bc2e03f40e3c3027cbe573c08b7b5134eef05070a973b40aed85d1eaab534"}
Nov 02 02:05:26.155101 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:26.155066    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:05:26.155317 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:26.155294    5875 patch_prober.go:29] interesting pod/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 02 02:05:26.155368 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:26.155347    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb" podUID="0931244f-232a-4588-8b26-eb3b5b5773c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 02 02:05:27.154950 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:27.154917    5875 patch_prober.go:29] interesting pod/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 02 02:05:27.155354 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:27.154980    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb" podUID="0931244f-232a-4588-8b26-eb3b5b5773c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 02 02:05:28.154759 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:28.154730    5875 patch_prober.go:29] interesting pod/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 02 02:05:28.154947 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:28.154789    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb" podUID="0931244f-232a-4588-8b26-eb3b5b5773c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 02 02:05:29.154600 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:29.154565    5875 patch_prober.go:29] interesting pod/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 02 02:05:29.155263 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:29.155209    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb" podUID="0931244f-232a-4588-8b26-eb3b5b5773c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 02 02:05:30.154502 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:30.154434    5875 patch_prober.go:29] interesting pod/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 02 02:05:30.154502 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:30.154493    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb" podUID="0931244f-232a-4588-8b26-eb3b5b5773c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 02 02:05:31.158683 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:31.158655    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:05:31.159175 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:31.159153    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:05:31.159774 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:31.159750    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:05:38.392067 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:38.392035    5875 scope.go:122] "RemoveContainer" containerID="aa0fed67b6e51dcf5abbb897243696b1eb384b1ab3ff5a5b4f04e311e0e50208"
Nov 02 02:05:41.322745 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:05:41.322660    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:05:51.979624 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:51.979588    5875 generic.go:356] "Generic (PLEG): container finished" podID="f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" containerID="ab0e4330b8ab6fc2f3d8cd2eb1bdf8d62ef21d2d23bd3bcc23dec820a8eddbab" exitCode=0
Nov 02 02:05:51.980095 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:51.979658    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9" event={"ID":"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e","Type":"ContainerDied","Data":"ab0e4330b8ab6fc2f3d8cd2eb1bdf8d62ef21d2d23bd3bcc23dec820a8eddbab"}
Nov 02 02:05:51.980095 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:51.980075    5875 scope.go:122] "RemoveContainer" containerID="ab0e4330b8ab6fc2f3d8cd2eb1bdf8d62ef21d2d23bd3bcc23dec820a8eddbab"
Nov 02 02:05:51.981878 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:51.981846    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:05:51.982012 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:51.981931    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:05:51.982012 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:51.981982    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:05:52.991563 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:52.991531    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9" event={"ID":"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e","Type":"ContainerStarted","Data":"b81ed9d044bfa0f851dae0ea861de42672d2af0932b168b1fe84fb38e843d954"}
Nov 02 02:05:52.993348 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:52.993323    5875 generic.go:356] "Generic (PLEG): container finished" podID="6c95abfc-9178-4ce3-9e78-cab75960a726" containerID="d9246068f333c1ef64192cb072d014ce4febee610a7cf888bd7c10d508f582aa" exitCode=0
Nov 02 02:05:52.993490 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:52.993404    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp" event={"ID":"6c95abfc-9178-4ce3-9e78-cab75960a726","Type":"ContainerDied","Data":"d9246068f333c1ef64192cb072d014ce4febee610a7cf888bd7c10d508f582aa"}
Nov 02 02:05:52.993839 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:52.993821    5875 scope.go:122] "RemoveContainer" containerID="d9246068f333c1ef64192cb072d014ce4febee610a7cf888bd7c10d508f582aa"
Nov 02 02:05:52.995574 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:52.995546    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:05:52.995646 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:52.995615    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:05:52.995686 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:52.995644    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:05:53.237244 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:53.236569    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:05:53.246017 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:53.245921    5875 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 02 02:05:53.246017 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:53.246002    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9" podUID="f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 02 02:05:54.064554 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:54.064516    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp" event={"ID":"6c95abfc-9178-4ce3-9e78-cab75960a726","Type":"ContainerStarted","Data":"673e1e5f47e5a6a577da54fe1a4c540a9ac1081f1e7bd5526ca3655a6fa4d060"}
Nov 02 02:05:54.236370 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:54.236328    5875 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 02 02:05:54.236664 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:54.236636    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9" podUID="f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 02 02:05:54.752600 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:54.752558    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:05:54.753882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:54.753854    5875 patch_prober.go:29] interesting pod/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 02 02:05:54.753965 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:54.753921    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp" podUID="6c95abfc-9178-4ce3-9e78-cab75960a726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 02 02:05:54.958971 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:54.958929    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:05:55.245247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:55.244895    5875 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 02 02:05:55.245247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:55.244951    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9" podUID="f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 02 02:05:55.753579 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:55.753542    5875 patch_prober.go:29] interesting pod/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 02 02:05:55.753782 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:55.753614    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp" podUID="6c95abfc-9178-4ce3-9e78-cab75960a726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 02 02:05:56.232677 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:56.232637    5875 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 02 02:05:56.232994 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:56.232965    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9" podUID="f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 02 02:05:56.753214 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:56.752150    5875 patch_prober.go:29] interesting pod/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 02 02:05:56.753214 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:56.752259    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp" podUID="6c95abfc-9178-4ce3-9e78-cab75960a726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 02 02:05:56.977422 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:56.977397    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:05:57.234684 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:57.234649    5875 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 02 02:05:57.234994 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:57.234965    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9" podUID="f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 02 02:05:57.751618 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:57.751587    5875 patch_prober.go:29] interesting pod/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 02 02:05:57.751870 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:57.751844    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp" podUID="6c95abfc-9178-4ce3-9e78-cab75960a726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 02 02:05:58.188433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:58.188322    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7"]
Nov 02 02:05:58.242460 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:58.242422    5875 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 02 02:05:58.243034 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:58.243004    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9" podUID="f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 02 02:05:58.347600 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:58.347558    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb"]
Nov 02 02:05:58.754828 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:58.753542    5875 patch_prober.go:29] interesting pod/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 02 02:05:58.754828 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:58.753601    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp" podUID="6c95abfc-9178-4ce3-9e78-cab75960a726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 02 02:05:59.237634 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:59.237603    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:05:59.238827 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:05:59.238804    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:06:00.130651 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:00.129631    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:06:00.135702 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:00.135612    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:06:03.188731 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:03.188665    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7" podUID="4b5be69e-c305-4530-8322-adf910287bfe" containerName="payload-processing" containerID="cri-o://37cd3f20f89ea14fc2bc6c0c92b208ea764cf3aeda9f5dbe30c308b102381f5a" gracePeriod=25
Nov 02 02:06:03.274386 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:06:03.274349    5875 logging.go:55] [core] [Channel #4064 SubChannel #4066]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.157:50052", ServerName: "10.134.0.157:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.157:50052: connect: connection refused"
Nov 02 02:06:03.348631 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:03.348592    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb" podUID="58d02a0f-00d6-4a8f-ac21-b473bd8a440e" containerName="payload-pre-processing" containerID="cri-o://00399438fcfcc16f9183593c759b66bb2ea56ae99cc068a3db2a5a40bb54c271" gracePeriod=25
Nov 02 02:06:03.484494 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:03.484468    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7"
Nov 02 02:06:03.580650 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:03.580605    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b5be69e-c305-4530-8322-adf910287bfe-kube-api-access-vcx9b\" (UniqueName: \"kubernetes.io/projected/4b5be69e-c305-4530-8322-adf910287bfe-kube-api-access-vcx9b\") pod \"4b5be69e-c305-4530-8322-adf910287bfe\" (UID: \"4b5be69e-c305-4530-8322-adf910287bfe\") "
Nov 02 02:06:03.580873 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:03.580854    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b5be69e-c305-4530-8322-adf910287bfe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b5be69e-c305-4530-8322-adf910287bfe-plugins-config-volume\") pod \"4b5be69e-c305-4530-8322-adf910287bfe\" (UID: \"4b5be69e-c305-4530-8322-adf910287bfe\") "
Nov 02 02:06:03.581507 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:03.581467    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b5be69e-c305-4530-8322-adf910287bfe-plugins-config-volume" pod "4b5be69e-c305-4530-8322-adf910287bfe" (UID: "4b5be69e-c305-4530-8322-adf910287bfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:06:03.583508 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:03.583472    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b5be69e-c305-4530-8322-adf910287bfe-kube-api-access-vcx9b" pod "4b5be69e-c305-4530-8322-adf910287bfe" (UID: "4b5be69e-c305-4530-8322-adf910287bfe"). InnerVolumeSpecName "kube-api-access-vcx9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:03.612424 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:03.612400    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb"
Nov 02 02:06:03.682754 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:03.682726    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58d02a0f-00d6-4a8f-ac21-b473bd8a440e-kube-api-access-f75qg\" (UniqueName: \"kubernetes.io/projected/58d02a0f-00d6-4a8f-ac21-b473bd8a440e-kube-api-access-f75qg\") pod \"58d02a0f-00d6-4a8f-ac21-b473bd8a440e\" (UID: \"58d02a0f-00d6-4a8f-ac21-b473bd8a440e\") "
Nov 02 02:06:03.682942 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:03.682793    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58d02a0f-00d6-4a8f-ac21-b473bd8a440e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58d02a0f-00d6-4a8f-ac21-b473bd8a440e-plugins-config-volume\") pod \"58d02a0f-00d6-4a8f-ac21-b473bd8a440e\" (UID: \"58d02a0f-00d6-4a8f-ac21-b473bd8a440e\") "
Nov 02 02:06:03.683187 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:03.683164    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d02a0f-00d6-4a8f-ac21-b473bd8a440e-plugins-config-volume" pod "58d02a0f-00d6-4a8f-ac21-b473bd8a440e" (UID: "58d02a0f-00d6-4a8f-ac21-b473bd8a440e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:06:03.683361 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:03.683345    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b5be69e-c305-4530-8322-adf910287bfe-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:03.683424 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:03.683367    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcx9b\" (UniqueName: \"kubernetes.io/projected/4b5be69e-c305-4530-8322-adf910287bfe-kube-api-access-vcx9b\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:03.683424 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:03.683382    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58d02a0f-00d6-4a8f-ac21-b473bd8a440e-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:03.684871 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:03.684849    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d02a0f-00d6-4a8f-ac21-b473bd8a440e-kube-api-access-f75qg" pod "58d02a0f-00d6-4a8f-ac21-b473bd8a440e" (UID: "58d02a0f-00d6-4a8f-ac21-b473bd8a440e"). InnerVolumeSpecName "kube-api-access-f75qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:03.784715 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:03.784684    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f75qg\" (UniqueName: \"kubernetes.io/projected/58d02a0f-00d6-4a8f-ac21-b473bd8a440e-kube-api-access-f75qg\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:04.181604 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:04.181518    5875 generic.go:356] "Generic (PLEG): container finished" podID="4b5be69e-c305-4530-8322-adf910287bfe" containerID="37cd3f20f89ea14fc2bc6c0c92b208ea764cf3aeda9f5dbe30c308b102381f5a" exitCode=0
Nov 02 02:06:04.181604 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:04.181576    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7"
Nov 02 02:06:04.181843 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:04.181604    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7" event={"ID":"4b5be69e-c305-4530-8322-adf910287bfe","Type":"ContainerDied","Data":"37cd3f20f89ea14fc2bc6c0c92b208ea764cf3aeda9f5dbe30c308b102381f5a"}
Nov 02 02:06:04.181843 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:04.181666    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7" event={"ID":"4b5be69e-c305-4530-8322-adf910287bfe","Type":"ContainerDied","Data":"53385cf3735fe116a7fab4f7db8d71c71f5a6a05b665178ee1fb74c98e8991c0"}
Nov 02 02:06:04.181843 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:04.181694    5875 scope.go:122] "RemoveContainer" containerID="37cd3f20f89ea14fc2bc6c0c92b208ea764cf3aeda9f5dbe30c308b102381f5a"
Nov 02 02:06:04.183059 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:04.183038    5875 generic.go:356] "Generic (PLEG): container finished" podID="58d02a0f-00d6-4a8f-ac21-b473bd8a440e" containerID="00399438fcfcc16f9183593c759b66bb2ea56ae99cc068a3db2a5a40bb54c271" exitCode=0
Nov 02 02:06:04.183194 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:04.183096    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb"
Nov 02 02:06:04.183194 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:04.183141    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb" event={"ID":"58d02a0f-00d6-4a8f-ac21-b473bd8a440e","Type":"ContainerDied","Data":"00399438fcfcc16f9183593c759b66bb2ea56ae99cc068a3db2a5a40bb54c271"}
Nov 02 02:06:04.183335 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:04.183193    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb" event={"ID":"58d02a0f-00d6-4a8f-ac21-b473bd8a440e","Type":"ContainerDied","Data":"562acb1f8ef2480ccdbe9b86062f22b443cc22976b87d94916962853e38acecc"}
Nov 02 02:06:04.194708 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:04.194544    5875 scope.go:122] "RemoveContainer" containerID="37cd3f20f89ea14fc2bc6c0c92b208ea764cf3aeda9f5dbe30c308b102381f5a"
Nov 02 02:06:04.194951 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:06:04.194822    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37cd3f20f89ea14fc2bc6c0c92b208ea764cf3aeda9f5dbe30c308b102381f5a\": container with ID starting with 37cd3f20f89ea14fc2bc6c0c92b208ea764cf3aeda9f5dbe30c308b102381f5a not found: ID does not exist" containerID="37cd3f20f89ea14fc2bc6c0c92b208ea764cf3aeda9f5dbe30c308b102381f5a"
Nov 02 02:06:04.194951 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:04.194845    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37cd3f20f89ea14fc2bc6c0c92b208ea764cf3aeda9f5dbe30c308b102381f5a"} err="failed to get container status \"37cd3f20f89ea14fc2bc6c0c92b208ea764cf3aeda9f5dbe30c308b102381f5a\": rpc error: code = NotFound desc = could not find container \"37cd3f20f89ea14fc2bc6c0c92b208ea764cf3aeda9f5dbe30c308b102381f5a\": container with ID starting with 37cd3f20f89ea14fc2bc6c0c92b208ea764cf3aeda9f5dbe30c308b102381f5a not found: ID does not exist"
Nov 02 02:06:04.194951 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:04.194862    5875 scope.go:122] "RemoveContainer" containerID="00399438fcfcc16f9183593c759b66bb2ea56ae99cc068a3db2a5a40bb54c271"
Nov 02 02:06:04.219894 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:04.219867    5875 scope.go:122] "RemoveContainer" containerID="00399438fcfcc16f9183593c759b66bb2ea56ae99cc068a3db2a5a40bb54c271"
Nov 02 02:06:04.220204 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:06:04.220188    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00399438fcfcc16f9183593c759b66bb2ea56ae99cc068a3db2a5a40bb54c271\": container with ID starting with 00399438fcfcc16f9183593c759b66bb2ea56ae99cc068a3db2a5a40bb54c271 not found: ID does not exist" containerID="00399438fcfcc16f9183593c759b66bb2ea56ae99cc068a3db2a5a40bb54c271"
Nov 02 02:06:04.220304 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:04.220214    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00399438fcfcc16f9183593c759b66bb2ea56ae99cc068a3db2a5a40bb54c271"} err="failed to get container status \"00399438fcfcc16f9183593c759b66bb2ea56ae99cc068a3db2a5a40bb54c271\": rpc error: code = NotFound desc = could not find container \"00399438fcfcc16f9183593c759b66bb2ea56ae99cc068a3db2a5a40bb54c271\": container with ID starting with 00399438fcfcc16f9183593c759b66bb2ea56ae99cc068a3db2a5a40bb54c271 not found: ID does not exist"
Nov 02 02:06:04.275082 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:04.275038    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7" podUID="4b5be69e-c305-4530-8322-adf910287bfe" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.157:50052\" within 1s: context deadline exceeded"
Nov 02 02:06:04.275288 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:06:04.275093    5875 logging.go:55] [core] [Channel #4064 SubChannel #4066]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.157:50052", ServerName: "10.134.0.157:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.157:50052: operation was canceled"
Nov 02 02:06:04.347474 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:04.347439    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb"]
Nov 02 02:06:04.350793 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:04.350770    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-abc85a-5bf5cc4455-8hbwb"]
Nov 02 02:06:04.459546 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:04.459518    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7"]
Nov 02 02:06:04.468957 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:04.468929    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-abc85a-7d765887f9-hxhw7"]
Nov 02 02:06:04.707547 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:04.707506    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b5be69e-c305-4530-8322-adf910287bfe" path="/var/lib/kubelet/pods/4b5be69e-c305-4530-8322-adf910287bfe/volumes"
Nov 02 02:06:04.708072 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:04.708057    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58d02a0f-00d6-4a8f-ac21-b473bd8a440e" path="/var/lib/kubelet/pods/58d02a0f-00d6-4a8f-ac21-b473bd8a440e/volumes"
Nov 02 02:06:07.804237 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:07.804200    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:06:11.346268 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:06:11.346211    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:06:19.321514 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:19.321476    5875 generic.go:356] "Generic (PLEG): container finished" podID="fda26851-8e30-489a-896d-73d9bcdd783f" containerID="4eff3eed579d2fc8030603ab4af2ef048a5dac16cfe3c875cdf4bf341d9b5d82" exitCode=0
Nov 02 02:06:19.321982 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:19.321532    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk" event={"ID":"fda26851-8e30-489a-896d-73d9bcdd783f","Type":"ContainerDied","Data":"4eff3eed579d2fc8030603ab4af2ef048a5dac16cfe3c875cdf4bf341d9b5d82"}
Nov 02 02:06:19.322032 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:19.322008    5875 scope.go:122] "RemoveContainer" containerID="4eff3eed579d2fc8030603ab4af2ef048a5dac16cfe3c875cdf4bf341d9b5d82"
Nov 02 02:06:19.323889 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:19.323857    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:06:19.323956 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:19.323933    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:06:19.324002 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:19.323975    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:06:20.335130 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:20.335096    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk" event={"ID":"fda26851-8e30-489a-896d-73d9bcdd783f","Type":"ContainerStarted","Data":"67335c33dd2e519c62740b1864d34bd18dee90f4324a08cc1f480badaa0982a1"}
Nov 02 02:06:21.243601 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:21.243567    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:06:21.244685 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:21.244663    5875 patch_prober.go:29] interesting pod/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 02 02:06:21.244741 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:21.244714    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk" podUID="fda26851-8e30-489a-896d-73d9bcdd783f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 02 02:06:22.243867 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:22.243829    5875 patch_prober.go:29] interesting pod/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 02 02:06:22.244316 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:22.243882    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk" podUID="fda26851-8e30-489a-896d-73d9bcdd783f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 02 02:06:23.244172 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:23.243636    5875 patch_prober.go:29] interesting pod/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 02 02:06:23.244172 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:23.243693    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk" podUID="fda26851-8e30-489a-896d-73d9bcdd783f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 02 02:06:23.377920 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:23.377889    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:06:24.243986 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:24.243946    5875 patch_prober.go:29] interesting pod/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 02 02:06:24.244181 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:24.244007    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk" podUID="fda26851-8e30-489a-896d-73d9bcdd783f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 02 02:06:25.244002 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:25.243963    5875 patch_prober.go:29] interesting pod/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 02 02:06:25.244475 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:25.244030    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk" podUID="fda26851-8e30-489a-896d-73d9bcdd783f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 02 02:06:26.248562 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:26.248528    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:06:26.249753 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:26.249730    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:06:37.796279 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:37.796241    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb"]
Nov 02 02:06:37.922920 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:37.922863    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz"]
Nov 02 02:06:38.248476 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:38.248432    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf"]
Nov 02 02:06:38.249871 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:38.249647    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf" podUID="4eda11d8-9d48-47e9-b94a-223847be8d73" containerName="maas-api" containerID="cri-o://eb2739915e8530a67db13dc1e602603a238fb87299d4f365961bd7b69ffdeb72" gracePeriod=30
Nov 02 02:06:38.451588 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:38.451555    5875 scope.go:122] "RemoveContainer" containerID="670725bdf24dece7cfba264f997b88c7b5f696dc9b7b2e20ee4a2b1842fa61ec"
Nov 02 02:06:38.522530 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:38.518820    5875 generic.go:356] "Generic (PLEG): container finished" podID="4eda11d8-9d48-47e9-b94a-223847be8d73" containerID="eb2739915e8530a67db13dc1e602603a238fb87299d4f365961bd7b69ffdeb72" exitCode=0
Nov 02 02:06:38.522530 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:38.518983    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf" event={"ID":"4eda11d8-9d48-47e9-b94a-223847be8d73","Type":"ContainerDied","Data":"eb2739915e8530a67db13dc1e602603a238fb87299d4f365961bd7b69ffdeb72"}
Nov 02 02:06:38.618361 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:38.618334    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf"
Nov 02 02:06:38.717183 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:38.713163    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eda11d8-9d48-47e9-b94a-223847be8d73-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eda11d8-9d48-47e9-b94a-223847be8d73-maas-api-tls\") pod \"4eda11d8-9d48-47e9-b94a-223847be8d73\" (UID: \"4eda11d8-9d48-47e9-b94a-223847be8d73\") "
Nov 02 02:06:38.717183 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:38.713296    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eda11d8-9d48-47e9-b94a-223847be8d73-kube-api-access-8qtr4\" (UniqueName: \"kubernetes.io/projected/4eda11d8-9d48-47e9-b94a-223847be8d73-kube-api-access-8qtr4\") pod \"4eda11d8-9d48-47e9-b94a-223847be8d73\" (UID: \"4eda11d8-9d48-47e9-b94a-223847be8d73\") "
Nov 02 02:06:38.717183 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:38.713435    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eda11d8-9d48-47e9-b94a-223847be8d73-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eda11d8-9d48-47e9-b94a-223847be8d73-metrics-certs\") pod \"4eda11d8-9d48-47e9-b94a-223847be8d73\" (UID: \"4eda11d8-9d48-47e9-b94a-223847be8d73\") "
Nov 02 02:06:38.717183 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:38.713517    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eda11d8-9d48-47e9-b94a-223847be8d73-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eda11d8-9d48-47e9-b94a-223847be8d73-openshift-service-ca\") pod \"4eda11d8-9d48-47e9-b94a-223847be8d73\" (UID: \"4eda11d8-9d48-47e9-b94a-223847be8d73\") "
Nov 02 02:06:38.718116 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:38.718086    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eda11d8-9d48-47e9-b94a-223847be8d73-openshift-service-ca" pod "4eda11d8-9d48-47e9-b94a-223847be8d73" (UID: "4eda11d8-9d48-47e9-b94a-223847be8d73"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:06:38.728540 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:38.728484    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eda11d8-9d48-47e9-b94a-223847be8d73-kube-api-access-8qtr4" pod "4eda11d8-9d48-47e9-b94a-223847be8d73" (UID: "4eda11d8-9d48-47e9-b94a-223847be8d73"). InnerVolumeSpecName "kube-api-access-8qtr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:38.739277 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:38.734421    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eda11d8-9d48-47e9-b94a-223847be8d73-metrics-certs" pod "4eda11d8-9d48-47e9-b94a-223847be8d73" (UID: "4eda11d8-9d48-47e9-b94a-223847be8d73"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:06:38.739277 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:38.738407    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eda11d8-9d48-47e9-b94a-223847be8d73-maas-api-tls" pod "4eda11d8-9d48-47e9-b94a-223847be8d73" (UID: "4eda11d8-9d48-47e9-b94a-223847be8d73"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:06:38.815664 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:38.815572    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qtr4\" (UniqueName: \"kubernetes.io/projected/4eda11d8-9d48-47e9-b94a-223847be8d73-kube-api-access-8qtr4\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:38.815664 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:38.815610    5875 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eda11d8-9d48-47e9-b94a-223847be8d73-metrics-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:38.815664 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:38.815627    5875 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eda11d8-9d48-47e9-b94a-223847be8d73-openshift-service-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:38.815664 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:38.815641    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eda11d8-9d48-47e9-b94a-223847be8d73-maas-api-tls\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:39.446357 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:39.446320    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm"]
Nov 02 02:06:39.446591 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:39.446565    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm" podUID="6b93b73a-53d8-463f-8f68-4490e7a8253f" containerName="maas-api" containerID="cri-o://fee85df8658ace5e0511e159c941ac7a92f9ad86fb52c40326e4dd6abb3a1b91" gracePeriod=30
Nov 02 02:06:39.528280 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:39.528201    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf" event={"ID":"4eda11d8-9d48-47e9-b94a-223847be8d73","Type":"ContainerDied","Data":"d04b286005bc55cf8651065936f7d79ed317f9289d36c417b1b66039c157d710"}
Nov 02 02:06:39.528498 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:39.528315    5875 scope.go:122] "RemoveContainer" containerID="eb2739915e8530a67db13dc1e602603a238fb87299d4f365961bd7b69ffdeb72"
Nov 02 02:06:39.528498 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:39.528479    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf"
Nov 02 02:06:39.773276 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:39.764439    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf"]
Nov 02 02:06:39.776164 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:39.776131    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bd3971-6848975575-8fwxf"]
Nov 02 02:06:39.850246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:39.846615    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm"
Nov 02 02:06:39.872247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:39.871113    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b93b73a-53d8-463f-8f68-4490e7a8253f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b93b73a-53d8-463f-8f68-4490e7a8253f-metrics-certs\") pod \"6b93b73a-53d8-463f-8f68-4490e7a8253f\" (UID: \"6b93b73a-53d8-463f-8f68-4490e7a8253f\") "
Nov 02 02:06:39.872247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:39.871191    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b93b73a-53d8-463f-8f68-4490e7a8253f-kube-api-access-zv4lw\" (UniqueName: \"kubernetes.io/projected/6b93b73a-53d8-463f-8f68-4490e7a8253f-kube-api-access-zv4lw\") pod \"6b93b73a-53d8-463f-8f68-4490e7a8253f\" (UID: \"6b93b73a-53d8-463f-8f68-4490e7a8253f\") "
Nov 02 02:06:39.872247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:39.871360    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b93b73a-53d8-463f-8f68-4490e7a8253f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b93b73a-53d8-463f-8f68-4490e7a8253f-openshift-service-ca\") pod \"6b93b73a-53d8-463f-8f68-4490e7a8253f\" (UID: \"6b93b73a-53d8-463f-8f68-4490e7a8253f\") "
Nov 02 02:06:39.872247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:39.871545    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b93b73a-53d8-463f-8f68-4490e7a8253f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b93b73a-53d8-463f-8f68-4490e7a8253f-maas-api-tls\") pod \"6b93b73a-53d8-463f-8f68-4490e7a8253f\" (UID: \"6b93b73a-53d8-463f-8f68-4490e7a8253f\") "
Nov 02 02:06:39.905247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:39.882846    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b93b73a-53d8-463f-8f68-4490e7a8253f-openshift-service-ca" pod "6b93b73a-53d8-463f-8f68-4490e7a8253f" (UID: "6b93b73a-53d8-463f-8f68-4490e7a8253f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:06:39.905247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:39.885965    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b93b73a-53d8-463f-8f68-4490e7a8253f-kube-api-access-zv4lw" pod "6b93b73a-53d8-463f-8f68-4490e7a8253f" (UID: "6b93b73a-53d8-463f-8f68-4490e7a8253f"). InnerVolumeSpecName "kube-api-access-zv4lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:39.905247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:39.890127    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b93b73a-53d8-463f-8f68-4490e7a8253f-metrics-certs" pod "6b93b73a-53d8-463f-8f68-4490e7a8253f" (UID: "6b93b73a-53d8-463f-8f68-4490e7a8253f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:06:39.906097 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:39.906045    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b93b73a-53d8-463f-8f68-4490e7a8253f-maas-api-tls" pod "6b93b73a-53d8-463f-8f68-4490e7a8253f" (UID: "6b93b73a-53d8-463f-8f68-4490e7a8253f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:06:39.969544 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:39.969498    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv"]
Nov 02 02:06:39.975392 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:39.975361    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b93b73a-53d8-463f-8f68-4490e7a8253f-maas-api-tls\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:39.975621 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:39.975606    5875 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b93b73a-53d8-463f-8f68-4490e7a8253f-metrics-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:39.975841 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:39.975825    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv4lw\" (UniqueName: \"kubernetes.io/projected/6b93b73a-53d8-463f-8f68-4490e7a8253f-kube-api-access-zv4lw\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:39.975937 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:39.975925    5875 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b93b73a-53d8-463f-8f68-4490e7a8253f-openshift-service-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:40.013689 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:40.013641    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729"]
Nov 02 02:06:40.578189 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:40.578146    5875 generic.go:356] "Generic (PLEG): container finished" podID="6b93b73a-53d8-463f-8f68-4490e7a8253f" containerID="fee85df8658ace5e0511e159c941ac7a92f9ad86fb52c40326e4dd6abb3a1b91" exitCode=0
Nov 02 02:06:40.578406 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:40.578197    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm" event={"ID":"6b93b73a-53d8-463f-8f68-4490e7a8253f","Type":"ContainerDied","Data":"fee85df8658ace5e0511e159c941ac7a92f9ad86fb52c40326e4dd6abb3a1b91"}
Nov 02 02:06:40.578406 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:40.578260    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm"
Nov 02 02:06:40.578406 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:40.578284    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm" event={"ID":"6b93b73a-53d8-463f-8f68-4490e7a8253f","Type":"ContainerDied","Data":"8412853f880f94bce7e87dd0ca627762fb9d3650b79daa1c1e776c50a901b7b3"}
Nov 02 02:06:40.578406 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:40.578319    5875 scope.go:122] "RemoveContainer" containerID="fee85df8658ace5e0511e159c941ac7a92f9ad86fb52c40326e4dd6abb3a1b91"
Nov 02 02:06:40.591013 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:40.590986    5875 scope.go:122] "RemoveContainer" containerID="fee85df8658ace5e0511e159c941ac7a92f9ad86fb52c40326e4dd6abb3a1b91"
Nov 02 02:06:40.591360 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:06:40.591336    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fee85df8658ace5e0511e159c941ac7a92f9ad86fb52c40326e4dd6abb3a1b91\": container with ID starting with fee85df8658ace5e0511e159c941ac7a92f9ad86fb52c40326e4dd6abb3a1b91 not found: ID does not exist" containerID="fee85df8658ace5e0511e159c941ac7a92f9ad86fb52c40326e4dd6abb3a1b91"
Nov 02 02:06:40.591466 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:40.591371    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fee85df8658ace5e0511e159c941ac7a92f9ad86fb52c40326e4dd6abb3a1b91"} err="failed to get container status \"fee85df8658ace5e0511e159c941ac7a92f9ad86fb52c40326e4dd6abb3a1b91\": rpc error: code = NotFound desc = could not find container \"fee85df8658ace5e0511e159c941ac7a92f9ad86fb52c40326e4dd6abb3a1b91\": container with ID starting with fee85df8658ace5e0511e159c941ac7a92f9ad86fb52c40326e4dd6abb3a1b91 not found: ID does not exist"
Nov 02 02:06:40.697447 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:40.697417    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eda11d8-9d48-47e9-b94a-223847be8d73" path="/var/lib/kubelet/pods/4eda11d8-9d48-47e9-b94a-223847be8d73/volumes"
Nov 02 02:06:40.743946 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:40.743903    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm"]
Nov 02 02:06:40.747812 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:40.747786    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7a24b9-6f6c9ddd67-hjqwm"]
Nov 02 02:06:41.323189 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:06:41.323151    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:06:42.697026 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:42.696990    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b93b73a-53d8-463f-8f68-4490e7a8253f" path="/var/lib/kubelet/pods/6b93b73a-53d8-463f-8f68-4490e7a8253f/volumes"
Nov 02 02:06:42.797182 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:42.797115    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb" podUID="0656b83a-c10d-4929-b029-c3852202b703" containerName="payload-processing" containerID="cri-o://dec47ffb64038d076330765c568a8f6a30b5137099e3b9cab444f477e7d04566" gracePeriod=25
Nov 02 02:06:42.924046 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:42.924003    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz" podUID="0d212576-5a9c-4523-8a0f-93b851be6406" containerName="payload-pre-processing" containerID="cri-o://435867d96f9fd47fd0e56cc38fd18a366d60441601ae422ad4a9388fd2327a7d" gracePeriod=25
Nov 02 02:06:43.088356 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.088322    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb"
Nov 02 02:06:43.157348 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.157315    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0656b83a-c10d-4929-b029-c3852202b703-kube-api-access-88gnl\" (UniqueName: \"kubernetes.io/projected/0656b83a-c10d-4929-b029-c3852202b703-kube-api-access-88gnl\") pod \"0656b83a-c10d-4929-b029-c3852202b703\" (UID: \"0656b83a-c10d-4929-b029-c3852202b703\") "
Nov 02 02:06:43.157509 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.157491    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0656b83a-c10d-4929-b029-c3852202b703-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0656b83a-c10d-4929-b029-c3852202b703-plugins-config-volume\") pod \"0656b83a-c10d-4929-b029-c3852202b703\" (UID: \"0656b83a-c10d-4929-b029-c3852202b703\") "
Nov 02 02:06:43.158112 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.158043    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0656b83a-c10d-4929-b029-c3852202b703-plugins-config-volume" pod "0656b83a-c10d-4929-b029-c3852202b703" (UID: "0656b83a-c10d-4929-b029-c3852202b703"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:06:43.158291 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.158268    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0656b83a-c10d-4929-b029-c3852202b703-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:43.160578 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.160546    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0656b83a-c10d-4929-b029-c3852202b703-kube-api-access-88gnl" pod "0656b83a-c10d-4929-b029-c3852202b703" (UID: "0656b83a-c10d-4929-b029-c3852202b703"). InnerVolumeSpecName "kube-api-access-88gnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:43.188451 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.188430    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz"
Nov 02 02:06:43.259360 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.259290    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d212576-5a9c-4523-8a0f-93b851be6406-kube-api-access-jz2j8\" (UniqueName: \"kubernetes.io/projected/0d212576-5a9c-4523-8a0f-93b851be6406-kube-api-access-jz2j8\") pod \"0d212576-5a9c-4523-8a0f-93b851be6406\" (UID: \"0d212576-5a9c-4523-8a0f-93b851be6406\") "
Nov 02 02:06:43.259494 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.259372    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d212576-5a9c-4523-8a0f-93b851be6406-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d212576-5a9c-4523-8a0f-93b851be6406-plugins-config-volume\") pod \"0d212576-5a9c-4523-8a0f-93b851be6406\" (UID: \"0d212576-5a9c-4523-8a0f-93b851be6406\") "
Nov 02 02:06:43.259723 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.259690    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d212576-5a9c-4523-8a0f-93b851be6406-plugins-config-volume" pod "0d212576-5a9c-4523-8a0f-93b851be6406" (UID: "0d212576-5a9c-4523-8a0f-93b851be6406"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:06:43.260020 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.260004    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d212576-5a9c-4523-8a0f-93b851be6406-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:43.260076 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.260024    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88gnl\" (UniqueName: \"kubernetes.io/projected/0656b83a-c10d-4929-b029-c3852202b703-kube-api-access-88gnl\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:43.261645 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.261623    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d212576-5a9c-4523-8a0f-93b851be6406-kube-api-access-jz2j8" pod "0d212576-5a9c-4523-8a0f-93b851be6406" (UID: "0d212576-5a9c-4523-8a0f-93b851be6406"). InnerVolumeSpecName "kube-api-access-jz2j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:43.361890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.361848    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz2j8\" (UniqueName: \"kubernetes.io/projected/0d212576-5a9c-4523-8a0f-93b851be6406-kube-api-access-jz2j8\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:43.631073 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.630995    5875 generic.go:356] "Generic (PLEG): container finished" podID="0d212576-5a9c-4523-8a0f-93b851be6406" containerID="435867d96f9fd47fd0e56cc38fd18a366d60441601ae422ad4a9388fd2327a7d" exitCode=0
Nov 02 02:06:43.631073 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.631044    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz"
Nov 02 02:06:43.631329 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.631082    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz" event={"ID":"0d212576-5a9c-4523-8a0f-93b851be6406","Type":"ContainerDied","Data":"435867d96f9fd47fd0e56cc38fd18a366d60441601ae422ad4a9388fd2327a7d"}
Nov 02 02:06:43.631329 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.631131    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz" event={"ID":"0d212576-5a9c-4523-8a0f-93b851be6406","Type":"ContainerDied","Data":"f29c1c61c860109e943dc8244c253f8770f3bc76312348d24d6acaf71bd07eb8"}
Nov 02 02:06:43.631329 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.631162    5875 scope.go:122] "RemoveContainer" containerID="435867d96f9fd47fd0e56cc38fd18a366d60441601ae422ad4a9388fd2327a7d"
Nov 02 02:06:43.632721 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.632700    5875 generic.go:356] "Generic (PLEG): container finished" podID="0656b83a-c10d-4929-b029-c3852202b703" containerID="dec47ffb64038d076330765c568a8f6a30b5137099e3b9cab444f477e7d04566" exitCode=0
Nov 02 02:06:43.632838 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.632790    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb"
Nov 02 02:06:43.632838 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.632827    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb" event={"ID":"0656b83a-c10d-4929-b029-c3852202b703","Type":"ContainerDied","Data":"dec47ffb64038d076330765c568a8f6a30b5137099e3b9cab444f477e7d04566"}
Nov 02 02:06:43.632943 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.632873    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb" event={"ID":"0656b83a-c10d-4929-b029-c3852202b703","Type":"ContainerDied","Data":"4bb6e3293299a2c76e226551470240fd588f53b56d14bacded6303a8a8ebf7eb"}
Nov 02 02:06:43.644517 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.644501    5875 scope.go:122] "RemoveContainer" containerID="435867d96f9fd47fd0e56cc38fd18a366d60441601ae422ad4a9388fd2327a7d"
Nov 02 02:06:43.644806 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:06:43.644786    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"435867d96f9fd47fd0e56cc38fd18a366d60441601ae422ad4a9388fd2327a7d\": container with ID starting with 435867d96f9fd47fd0e56cc38fd18a366d60441601ae422ad4a9388fd2327a7d not found: ID does not exist" containerID="435867d96f9fd47fd0e56cc38fd18a366d60441601ae422ad4a9388fd2327a7d"
Nov 02 02:06:43.644895 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.644817    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"435867d96f9fd47fd0e56cc38fd18a366d60441601ae422ad4a9388fd2327a7d"} err="failed to get container status \"435867d96f9fd47fd0e56cc38fd18a366d60441601ae422ad4a9388fd2327a7d\": rpc error: code = NotFound desc = could not find container \"435867d96f9fd47fd0e56cc38fd18a366d60441601ae422ad4a9388fd2327a7d\": container with ID starting with 435867d96f9fd47fd0e56cc38fd18a366d60441601ae422ad4a9388fd2327a7d not found: ID does not exist"
Nov 02 02:06:43.644895 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.644838    5875 scope.go:122] "RemoveContainer" containerID="dec47ffb64038d076330765c568a8f6a30b5137099e3b9cab444f477e7d04566"
Nov 02 02:06:43.659417 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.659393    5875 scope.go:122] "RemoveContainer" containerID="dec47ffb64038d076330765c568a8f6a30b5137099e3b9cab444f477e7d04566"
Nov 02 02:06:43.659703 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:06:43.659682    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dec47ffb64038d076330765c568a8f6a30b5137099e3b9cab444f477e7d04566\": container with ID starting with dec47ffb64038d076330765c568a8f6a30b5137099e3b9cab444f477e7d04566 not found: ID does not exist" containerID="dec47ffb64038d076330765c568a8f6a30b5137099e3b9cab444f477e7d04566"
Nov 02 02:06:43.659764 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.659710    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec47ffb64038d076330765c568a8f6a30b5137099e3b9cab444f477e7d04566"} err="failed to get container status \"dec47ffb64038d076330765c568a8f6a30b5137099e3b9cab444f477e7d04566\": rpc error: code = NotFound desc = could not find container \"dec47ffb64038d076330765c568a8f6a30b5137099e3b9cab444f477e7d04566\": container with ID starting with dec47ffb64038d076330765c568a8f6a30b5137099e3b9cab444f477e7d04566 not found: ID does not exist"
Nov 02 02:06:43.799789 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.799761    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz"]
Nov 02 02:06:43.848970 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.848935    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c1aad5-d9bf4d78c-hbjsz"]
Nov 02 02:06:43.914462 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.914392    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb"]
Nov 02 02:06:43.919326 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:43.919298    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c1aad5-cf4bc8b6c-9v9lb"]
Nov 02 02:06:44.698442 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:44.698411    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0656b83a-c10d-4929-b029-c3852202b703" path="/var/lib/kubelet/pods/0656b83a-c10d-4929-b029-c3852202b703/volumes"
Nov 02 02:06:44.698859 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:44.698844    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d212576-5a9c-4523-8a0f-93b851be6406" path="/var/lib/kubelet/pods/0d212576-5a9c-4523-8a0f-93b851be6406/volumes"
Nov 02 02:06:44.971098 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:44.971049    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv" podUID="f6d65404-5fe2-41e1-b878-634a13c47eb8" containerName="payload-processing" containerID="cri-o://a85a74d16a85bc6c18cc1125c01b49356238376a4f4953c844655838b7764f2d" gracePeriod=25
Nov 02 02:06:45.014032 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.013997    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729" podUID="a587ba49-1db6-49fb-9b21-9a00b9eb504f" containerName="payload-pre-processing" containerID="cri-o://a8e1fd997e7a92ee40a951feb7c33acf8a2952a4042079dce901cc4451481919" gracePeriod=25
Nov 02 02:06:45.094317 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:06:45.094202    5875 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda587ba49_1db6_49fb_9b21_9a00b9eb504f.slice/crio-a8e1fd997e7a92ee40a951feb7c33acf8a2952a4042079dce901cc4451481919.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda587ba49_1db6_49fb_9b21_9a00b9eb504f.slice/crio-conmon-a8e1fd997e7a92ee40a951feb7c33acf8a2952a4042079dce901cc4451481919.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 02:06:45.319124 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.319081    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv"
Nov 02 02:06:45.345158 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.345131    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729"
Nov 02 02:06:45.392091 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.392069    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6d65404-5fe2-41e1-b878-634a13c47eb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6d65404-5fe2-41e1-b878-634a13c47eb8-plugins-config-volume\") pod \"f6d65404-5fe2-41e1-b878-634a13c47eb8\" (UID: \"f6d65404-5fe2-41e1-b878-634a13c47eb8\") "
Nov 02 02:06:45.392318 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.392304    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6d65404-5fe2-41e1-b878-634a13c47eb8-kube-api-access-92ksk\" (UniqueName: \"kubernetes.io/projected/f6d65404-5fe2-41e1-b878-634a13c47eb8-kube-api-access-92ksk\") pod \"f6d65404-5fe2-41e1-b878-634a13c47eb8\" (UID: \"f6d65404-5fe2-41e1-b878-634a13c47eb8\") "
Nov 02 02:06:45.392465 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.392427    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6d65404-5fe2-41e1-b878-634a13c47eb8-plugins-config-volume" pod "f6d65404-5fe2-41e1-b878-634a13c47eb8" (UID: "f6d65404-5fe2-41e1-b878-634a13c47eb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:06:45.392742 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.392726    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6d65404-5fe2-41e1-b878-634a13c47eb8-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:45.394454 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.394430    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d65404-5fe2-41e1-b878-634a13c47eb8-kube-api-access-92ksk" pod "f6d65404-5fe2-41e1-b878-634a13c47eb8" (UID: "f6d65404-5fe2-41e1-b878-634a13c47eb8"). InnerVolumeSpecName "kube-api-access-92ksk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:45.494088 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.494054    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a587ba49-1db6-49fb-9b21-9a00b9eb504f-kube-api-access-swzrx\" (UniqueName: \"kubernetes.io/projected/a587ba49-1db6-49fb-9b21-9a00b9eb504f-kube-api-access-swzrx\") pod \"a587ba49-1db6-49fb-9b21-9a00b9eb504f\" (UID: \"a587ba49-1db6-49fb-9b21-9a00b9eb504f\") "
Nov 02 02:06:45.494287 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.494167    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a587ba49-1db6-49fb-9b21-9a00b9eb504f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a587ba49-1db6-49fb-9b21-9a00b9eb504f-plugins-config-volume\") pod \"a587ba49-1db6-49fb-9b21-9a00b9eb504f\" (UID: \"a587ba49-1db6-49fb-9b21-9a00b9eb504f\") "
Nov 02 02:06:45.494526 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.494501    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a587ba49-1db6-49fb-9b21-9a00b9eb504f-plugins-config-volume" pod "a587ba49-1db6-49fb-9b21-9a00b9eb504f" (UID: "a587ba49-1db6-49fb-9b21-9a00b9eb504f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:06:45.494822 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.494808    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a587ba49-1db6-49fb-9b21-9a00b9eb504f-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:45.494866 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.494826    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92ksk\" (UniqueName: \"kubernetes.io/projected/f6d65404-5fe2-41e1-b878-634a13c47eb8-kube-api-access-92ksk\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:45.496386 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.496360    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a587ba49-1db6-49fb-9b21-9a00b9eb504f-kube-api-access-swzrx" pod "a587ba49-1db6-49fb-9b21-9a00b9eb504f" (UID: "a587ba49-1db6-49fb-9b21-9a00b9eb504f"). InnerVolumeSpecName "kube-api-access-swzrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:45.596286 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.596164    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swzrx\" (UniqueName: \"kubernetes.io/projected/a587ba49-1db6-49fb-9b21-9a00b9eb504f-kube-api-access-swzrx\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:45.678083 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.678047    5875 generic.go:356] "Generic (PLEG): container finished" podID="a587ba49-1db6-49fb-9b21-9a00b9eb504f" containerID="a8e1fd997e7a92ee40a951feb7c33acf8a2952a4042079dce901cc4451481919" exitCode=0
Nov 02 02:06:45.678321 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.678088    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729"
Nov 02 02:06:45.678321 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.678146    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729" event={"ID":"a587ba49-1db6-49fb-9b21-9a00b9eb504f","Type":"ContainerDied","Data":"a8e1fd997e7a92ee40a951feb7c33acf8a2952a4042079dce901cc4451481919"}
Nov 02 02:06:45.678321 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.678208    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729" event={"ID":"a587ba49-1db6-49fb-9b21-9a00b9eb504f","Type":"ContainerDied","Data":"7ae036d3b509905e2a08ca29f55f59ea39ced3aecbd11f605967dafbe518cbca"}
Nov 02 02:06:45.678321 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.678261    5875 scope.go:122] "RemoveContainer" containerID="a8e1fd997e7a92ee40a951feb7c33acf8a2952a4042079dce901cc4451481919"
Nov 02 02:06:45.679721 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.679696    5875 generic.go:356] "Generic (PLEG): container finished" podID="f6d65404-5fe2-41e1-b878-634a13c47eb8" containerID="a85a74d16a85bc6c18cc1125c01b49356238376a4f4953c844655838b7764f2d" exitCode=0
Nov 02 02:06:45.679855 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.679759    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv"
Nov 02 02:06:45.679855 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.679781    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv" event={"ID":"f6d65404-5fe2-41e1-b878-634a13c47eb8","Type":"ContainerDied","Data":"a85a74d16a85bc6c18cc1125c01b49356238376a4f4953c844655838b7764f2d"}
Nov 02 02:06:45.679855 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.679836    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv" event={"ID":"f6d65404-5fe2-41e1-b878-634a13c47eb8","Type":"ContainerDied","Data":"41f9ec88887ec464d913cbd3ea4cf7ffb84335aec1756fa60f5d9976dae24a05"}
Nov 02 02:06:45.691110 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.691094    5875 scope.go:122] "RemoveContainer" containerID="a8e1fd997e7a92ee40a951feb7c33acf8a2952a4042079dce901cc4451481919"
Nov 02 02:06:45.691450 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:06:45.691429    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e1fd997e7a92ee40a951feb7c33acf8a2952a4042079dce901cc4451481919\": container with ID starting with a8e1fd997e7a92ee40a951feb7c33acf8a2952a4042079dce901cc4451481919 not found: ID does not exist" containerID="a8e1fd997e7a92ee40a951feb7c33acf8a2952a4042079dce901cc4451481919"
Nov 02 02:06:45.691494 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.691471    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e1fd997e7a92ee40a951feb7c33acf8a2952a4042079dce901cc4451481919"} err="failed to get container status \"a8e1fd997e7a92ee40a951feb7c33acf8a2952a4042079dce901cc4451481919\": rpc error: code = NotFound desc = could not find container \"a8e1fd997e7a92ee40a951feb7c33acf8a2952a4042079dce901cc4451481919\": container with ID starting with a8e1fd997e7a92ee40a951feb7c33acf8a2952a4042079dce901cc4451481919 not found: ID does not exist"
Nov 02 02:06:45.691533 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.691492    5875 scope.go:122] "RemoveContainer" containerID="a85a74d16a85bc6c18cc1125c01b49356238376a4f4953c844655838b7764f2d"
Nov 02 02:06:45.702105 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.702066    5875 scope.go:122] "RemoveContainer" containerID="a85a74d16a85bc6c18cc1125c01b49356238376a4f4953c844655838b7764f2d"
Nov 02 02:06:45.702435 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:06:45.702413    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a85a74d16a85bc6c18cc1125c01b49356238376a4f4953c844655838b7764f2d\": container with ID starting with a85a74d16a85bc6c18cc1125c01b49356238376a4f4953c844655838b7764f2d not found: ID does not exist" containerID="a85a74d16a85bc6c18cc1125c01b49356238376a4f4953c844655838b7764f2d"
Nov 02 02:06:45.702514 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.702443    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a85a74d16a85bc6c18cc1125c01b49356238376a4f4953c844655838b7764f2d"} err="failed to get container status \"a85a74d16a85bc6c18cc1125c01b49356238376a4f4953c844655838b7764f2d\": rpc error: code = NotFound desc = could not find container \"a85a74d16a85bc6c18cc1125c01b49356238376a4f4953c844655838b7764f2d\": container with ID starting with a85a74d16a85bc6c18cc1125c01b49356238376a4f4953c844655838b7764f2d not found: ID does not exist"
Nov 02 02:06:45.878374 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.878281    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv"]
Nov 02 02:06:45.912282 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.912250    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7a24b9-6f675f97f5-p64sv"]
Nov 02 02:06:45.987902 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:45.987870    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729"]
Nov 02 02:06:46.001285 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:46.000655    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7a24b9-5d867894ff-9g729"]
Nov 02 02:06:46.696466 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:46.696438    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a587ba49-1db6-49fb-9b21-9a00b9eb504f" path="/var/lib/kubelet/pods/a587ba49-1db6-49fb-9b21-9a00b9eb504f/volumes"
Nov 02 02:06:46.696854 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:46.696839    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6d65404-5fe2-41e1-b878-634a13c47eb8" path="/var/lib/kubelet/pods/f6d65404-5fe2-41e1-b878-634a13c47eb8/volumes"
Nov 02 02:06:48.028524 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:48.028488    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"]
Nov 02 02:06:48.028913 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:48.028697    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx" podUID="160fc571-588a-47fd-9acc-f82010429400" containerName="istio-proxy" containerID="cri-o://172403de64910ad436e5c4b34066e68cf28625d4bb802cc3cbeef7e6c0b97a6b" gracePeriod=30
Nov 02 02:06:52.271275 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:52.271155    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"]
Nov 02 02:06:52.271825 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:52.271415    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp" podUID="6c95abfc-9178-4ce3-9e78-cab75960a726" containerName="istio-proxy" containerID="cri-o://673e1e5f47e5a6a577da54fe1a4c540a9ac1081f1e7bd5526ca3655a6fa4d060" gracePeriod=30
Nov 02 02:06:53.310140 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.310084    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:06:53.391053 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.391022    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-workload-socket\") pod \"160fc571-588a-47fd-9acc-f82010429400\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") "
Nov 02 02:06:53.391264 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.391060    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/160fc571-588a-47fd-9acc-f82010429400-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160fc571-588a-47fd-9acc-f82010429400-istiod-ca-cert\") pod \"160fc571-588a-47fd-9acc-f82010429400\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") "
Nov 02 02:06:53.391264 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.391098    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-istio-envoy\") pod \"160fc571-588a-47fd-9acc-f82010429400\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") "
Nov 02 02:06:53.391264 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.391162    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/160fc571-588a-47fd-9acc-f82010429400-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160fc571-588a-47fd-9acc-f82010429400-istio-podinfo\") pod \"160fc571-588a-47fd-9acc-f82010429400\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") "
Nov 02 02:06:53.391521 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.391352    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-workload-socket" pod "160fc571-588a-47fd-9acc-f82010429400" (UID: "160fc571-588a-47fd-9acc-f82010429400"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:53.391521 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.391417    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/160fc571-588a-47fd-9acc-f82010429400-kube-api-access-8nx4p\" (UniqueName: \"kubernetes.io/projected/160fc571-588a-47fd-9acc-f82010429400-kube-api-access-8nx4p\") pod \"160fc571-588a-47fd-9acc-f82010429400\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") "
Nov 02 02:06:53.391521 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.391478    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/160fc571-588a-47fd-9acc-f82010429400-istio-token\" (UniqueName: \"kubernetes.io/projected/160fc571-588a-47fd-9acc-f82010429400-istio-token\") pod \"160fc571-588a-47fd-9acc-f82010429400\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") "
Nov 02 02:06:53.391692 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.391532    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/160fc571-588a-47fd-9acc-f82010429400-istiod-ca-cert" pod "160fc571-588a-47fd-9acc-f82010429400" (UID: "160fc571-588a-47fd-9acc-f82010429400"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:06:53.391692 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.391546    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-istio-data\") pod \"160fc571-588a-47fd-9acc-f82010429400\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") "
Nov 02 02:06:53.391692 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.391592    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-credential-socket\") pod \"160fc571-588a-47fd-9acc-f82010429400\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") "
Nov 02 02:06:53.391692 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.391659    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-workload-certs\") pod \"160fc571-588a-47fd-9acc-f82010429400\" (UID: \"160fc571-588a-47fd-9acc-f82010429400\") "
Nov 02 02:06:53.391920 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.391799    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-istio-data" pod "160fc571-588a-47fd-9acc-f82010429400" (UID: "160fc571-588a-47fd-9acc-f82010429400"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:53.392186 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.392154    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-workload-certs" pod "160fc571-588a-47fd-9acc-f82010429400" (UID: "160fc571-588a-47fd-9acc-f82010429400"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:53.392475 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.392451    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:53.392475 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.392481    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160fc571-588a-47fd-9acc-f82010429400-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:53.392639 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.392496    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:53.392639 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.392511    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:53.392639 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.392514    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-credential-socket" pod "160fc571-588a-47fd-9acc-f82010429400" (UID: "160fc571-588a-47fd-9acc-f82010429400"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:53.393618 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.393591    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-istio-envoy" pod "160fc571-588a-47fd-9acc-f82010429400" (UID: "160fc571-588a-47fd-9acc-f82010429400"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:53.393618 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.393607    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/160fc571-588a-47fd-9acc-f82010429400-istio-podinfo" pod "160fc571-588a-47fd-9acc-f82010429400" (UID: "160fc571-588a-47fd-9acc-f82010429400"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:06:53.393750 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.393693    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160fc571-588a-47fd-9acc-f82010429400-kube-api-access-8nx4p" pod "160fc571-588a-47fd-9acc-f82010429400" (UID: "160fc571-588a-47fd-9acc-f82010429400"). InnerVolumeSpecName "kube-api-access-8nx4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:53.394033 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.394007    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160fc571-588a-47fd-9acc-f82010429400-istio-token" pod "160fc571-588a-47fd-9acc-f82010429400" (UID: "160fc571-588a-47fd-9acc-f82010429400"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:53.494719 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.494683    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nx4p\" (UniqueName: \"kubernetes.io/projected/160fc571-588a-47fd-9acc-f82010429400-kube-api-access-8nx4p\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:53.494719 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.494722    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/160fc571-588a-47fd-9acc-f82010429400-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:53.494946 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.494738    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:53.494946 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.494751    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160fc571-588a-47fd-9acc-f82010429400-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:53.494946 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.494767    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160fc571-588a-47fd-9acc-f82010429400-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:53.795203 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.795170    5875 generic.go:356] "Generic (PLEG): container finished" podID="160fc571-588a-47fd-9acc-f82010429400" containerID="172403de64910ad436e5c4b34066e68cf28625d4bb802cc3cbeef7e6c0b97a6b" exitCode=0
Nov 02 02:06:53.795404 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.795264    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"
Nov 02 02:06:53.795404 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.795263    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx" event={"ID":"160fc571-588a-47fd-9acc-f82010429400","Type":"ContainerDied","Data":"172403de64910ad436e5c4b34066e68cf28625d4bb802cc3cbeef7e6c0b97a6b"}
Nov 02 02:06:53.795404 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.795328    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx" event={"ID":"160fc571-588a-47fd-9acc-f82010429400","Type":"ContainerDied","Data":"b690303149e35a1cebd34bf1133c28dc6319f5b8bb261fbf947df5ca0c91ddef"}
Nov 02 02:06:53.795404 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.795360    5875 scope.go:122] "RemoveContainer" containerID="172403de64910ad436e5c4b34066e68cf28625d4bb802cc3cbeef7e6c0b97a6b"
Nov 02 02:06:53.807201 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.807184    5875 scope.go:122] "RemoveContainer" containerID="7b07b76c1ddf436f96965edd717d17da1e2483fbb0771470eda6cd9f91b68295"
Nov 02 02:06:53.816708 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.816689    5875 scope.go:122] "RemoveContainer" containerID="172403de64910ad436e5c4b34066e68cf28625d4bb802cc3cbeef7e6c0b97a6b"
Nov 02 02:06:53.816971 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:06:53.816943    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"172403de64910ad436e5c4b34066e68cf28625d4bb802cc3cbeef7e6c0b97a6b\": container with ID starting with 172403de64910ad436e5c4b34066e68cf28625d4bb802cc3cbeef7e6c0b97a6b not found: ID does not exist" containerID="172403de64910ad436e5c4b34066e68cf28625d4bb802cc3cbeef7e6c0b97a6b"
Nov 02 02:06:53.817078 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.816974    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"172403de64910ad436e5c4b34066e68cf28625d4bb802cc3cbeef7e6c0b97a6b"} err="failed to get container status \"172403de64910ad436e5c4b34066e68cf28625d4bb802cc3cbeef7e6c0b97a6b\": rpc error: code = NotFound desc = could not find container \"172403de64910ad436e5c4b34066e68cf28625d4bb802cc3cbeef7e6c0b97a6b\": container with ID starting with 172403de64910ad436e5c4b34066e68cf28625d4bb802cc3cbeef7e6c0b97a6b not found: ID does not exist"
Nov 02 02:06:53.817078 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.816993    5875 scope.go:122] "RemoveContainer" containerID="7b07b76c1ddf436f96965edd717d17da1e2483fbb0771470eda6cd9f91b68295"
Nov 02 02:06:53.817280 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:06:53.817257    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b07b76c1ddf436f96965edd717d17da1e2483fbb0771470eda6cd9f91b68295\": container with ID starting with 7b07b76c1ddf436f96965edd717d17da1e2483fbb0771470eda6cd9f91b68295 not found: ID does not exist" containerID="7b07b76c1ddf436f96965edd717d17da1e2483fbb0771470eda6cd9f91b68295"
Nov 02 02:06:53.817348 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.817289    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b07b76c1ddf436f96965edd717d17da1e2483fbb0771470eda6cd9f91b68295"} err="failed to get container status \"7b07b76c1ddf436f96965edd717d17da1e2483fbb0771470eda6cd9f91b68295\": rpc error: code = NotFound desc = could not find container \"7b07b76c1ddf436f96965edd717d17da1e2483fbb0771470eda6cd9f91b68295\": container with ID starting with 7b07b76c1ddf436f96965edd717d17da1e2483fbb0771470eda6cd9f91b68295 not found: ID does not exist"
Nov 02 02:06:53.977236 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.977198    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"]
Nov 02 02:06:53.981236 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:53.981201    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-bd3971-openshift-default-5d5c88b8b5-h6jpx"]
Nov 02 02:06:54.699712 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:54.699682    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="160fc571-588a-47fd-9acc-f82010429400" path="/var/lib/kubelet/pods/160fc571-588a-47fd-9acc-f82010429400/volumes"
Nov 02 02:06:57.661969 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.661934    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:06:57.673132 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.673081    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-token\" (UniqueName: \"kubernetes.io/projected/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-token\") pod \"6c95abfc-9178-4ce3-9e78-cab75960a726\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") "
Nov 02 02:06:57.673366 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.673169    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c95abfc-9178-4ce3-9e78-cab75960a726-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c95abfc-9178-4ce3-9e78-cab75960a726-istiod-ca-cert\") pod \"6c95abfc-9178-4ce3-9e78-cab75960a726\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") "
Nov 02 02:06:57.673366 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.673247    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-credential-socket\") pod \"6c95abfc-9178-4ce3-9e78-cab75960a726\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") "
Nov 02 02:06:57.673366 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.673319    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-podinfo\") pod \"6c95abfc-9178-4ce3-9e78-cab75960a726\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") "
Nov 02 02:06:57.673366 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.673361    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-envoy\") pod \"6c95abfc-9178-4ce3-9e78-cab75960a726\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") "
Nov 02 02:06:57.673605 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.673409    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-workload-socket\") pod \"6c95abfc-9178-4ce3-9e78-cab75960a726\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") "
Nov 02 02:06:57.673605 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.673452    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c95abfc-9178-4ce3-9e78-cab75960a726-kube-api-access-l7w8c\" (UniqueName: \"kubernetes.io/projected/6c95abfc-9178-4ce3-9e78-cab75960a726-kube-api-access-l7w8c\") pod \"6c95abfc-9178-4ce3-9e78-cab75960a726\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") "
Nov 02 02:06:57.673605 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.673557    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-data\") pod \"6c95abfc-9178-4ce3-9e78-cab75960a726\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") "
Nov 02 02:06:57.673772 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.673613    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-workload-certs\") pod \"6c95abfc-9178-4ce3-9e78-cab75960a726\" (UID: \"6c95abfc-9178-4ce3-9e78-cab75960a726\") "
Nov 02 02:06:57.674332 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.674307    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-workload-certs" pod "6c95abfc-9178-4ce3-9e78-cab75960a726" (UID: "6c95abfc-9178-4ce3-9e78-cab75960a726"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:57.674611 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.674582    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-workload-socket" pod "6c95abfc-9178-4ce3-9e78-cab75960a726" (UID: "6c95abfc-9178-4ce3-9e78-cab75960a726"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:57.674909 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.674882    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-credential-socket" pod "6c95abfc-9178-4ce3-9e78-cab75960a726" (UID: "6c95abfc-9178-4ce3-9e78-cab75960a726"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:57.675107 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.675075    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-data" pod "6c95abfc-9178-4ce3-9e78-cab75960a726" (UID: "6c95abfc-9178-4ce3-9e78-cab75960a726"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:57.676487 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.676456    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c95abfc-9178-4ce3-9e78-cab75960a726-istiod-ca-cert" pod "6c95abfc-9178-4ce3-9e78-cab75960a726" (UID: "6c95abfc-9178-4ce3-9e78-cab75960a726"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:06:57.678818 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.678789    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-envoy" pod "6c95abfc-9178-4ce3-9e78-cab75960a726" (UID: "6c95abfc-9178-4ce3-9e78-cab75960a726"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:06:57.679149 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.679121    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c95abfc-9178-4ce3-9e78-cab75960a726-kube-api-access-l7w8c" pod "6c95abfc-9178-4ce3-9e78-cab75960a726" (UID: "6c95abfc-9178-4ce3-9e78-cab75960a726"). InnerVolumeSpecName "kube-api-access-l7w8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:57.694338 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.694307    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-token" pod "6c95abfc-9178-4ce3-9e78-cab75960a726" (UID: "6c95abfc-9178-4ce3-9e78-cab75960a726"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:06:57.694560 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.694543    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-podinfo" pod "6c95abfc-9178-4ce3-9e78-cab75960a726" (UID: "6c95abfc-9178-4ce3-9e78-cab75960a726"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:06:57.775026 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.774997    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:57.775026 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.775021    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7w8c\" (UniqueName: \"kubernetes.io/projected/6c95abfc-9178-4ce3-9e78-cab75960a726-kube-api-access-l7w8c\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:57.775026 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.775031    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:57.775324 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.775040    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:57.775324 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.775051    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:57.775324 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.775059    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c95abfc-9178-4ce3-9e78-cab75960a726-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:57.775324 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.775066    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:57.775324 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.775075    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:57.775324 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.775083    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c95abfc-9178-4ce3-9e78-cab75960a726-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:06:57.832773 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.832635    5875 generic.go:356] "Generic (PLEG): container finished" podID="6c95abfc-9178-4ce3-9e78-cab75960a726" containerID="673e1e5f47e5a6a577da54fe1a4c540a9ac1081f1e7bd5526ca3655a6fa4d060" exitCode=0
Nov 02 02:06:57.832773 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.832700    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"
Nov 02 02:06:57.833155 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.832789    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp" event={"ID":"6c95abfc-9178-4ce3-9e78-cab75960a726","Type":"ContainerDied","Data":"673e1e5f47e5a6a577da54fe1a4c540a9ac1081f1e7bd5526ca3655a6fa4d060"}
Nov 02 02:06:57.833155 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.832835    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp" event={"ID":"6c95abfc-9178-4ce3-9e78-cab75960a726","Type":"ContainerDied","Data":"d290222e5de1c4b196b2d474d5cee7f40dac44d868b427ab09d02cba0837b25e"}
Nov 02 02:06:57.833155 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.832862    5875 scope.go:122] "RemoveContainer" containerID="673e1e5f47e5a6a577da54fe1a4c540a9ac1081f1e7bd5526ca3655a6fa4d060"
Nov 02 02:06:57.846398 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.846373    5875 scope.go:122] "RemoveContainer" containerID="d9246068f333c1ef64192cb072d014ce4febee610a7cf888bd7c10d508f582aa"
Nov 02 02:06:57.862386 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.862360    5875 scope.go:122] "RemoveContainer" containerID="673e1e5f47e5a6a577da54fe1a4c540a9ac1081f1e7bd5526ca3655a6fa4d060"
Nov 02 02:06:57.863800 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:06:57.863768    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"673e1e5f47e5a6a577da54fe1a4c540a9ac1081f1e7bd5526ca3655a6fa4d060\": container with ID starting with 673e1e5f47e5a6a577da54fe1a4c540a9ac1081f1e7bd5526ca3655a6fa4d060 not found: ID does not exist" containerID="673e1e5f47e5a6a577da54fe1a4c540a9ac1081f1e7bd5526ca3655a6fa4d060"
Nov 02 02:06:57.863929 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.863811    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"673e1e5f47e5a6a577da54fe1a4c540a9ac1081f1e7bd5526ca3655a6fa4d060"} err="failed to get container status \"673e1e5f47e5a6a577da54fe1a4c540a9ac1081f1e7bd5526ca3655a6fa4d060\": rpc error: code = NotFound desc = could not find container \"673e1e5f47e5a6a577da54fe1a4c540a9ac1081f1e7bd5526ca3655a6fa4d060\": container with ID starting with 673e1e5f47e5a6a577da54fe1a4c540a9ac1081f1e7bd5526ca3655a6fa4d060 not found: ID does not exist"
Nov 02 02:06:57.863929 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.863840    5875 scope.go:122] "RemoveContainer" containerID="d9246068f333c1ef64192cb072d014ce4febee610a7cf888bd7c10d508f582aa"
Nov 02 02:06:57.864772 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:06:57.864733    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9246068f333c1ef64192cb072d014ce4febee610a7cf888bd7c10d508f582aa\": container with ID starting with d9246068f333c1ef64192cb072d014ce4febee610a7cf888bd7c10d508f582aa not found: ID does not exist" containerID="d9246068f333c1ef64192cb072d014ce4febee610a7cf888bd7c10d508f582aa"
Nov 02 02:06:57.864865 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:57.864780    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9246068f333c1ef64192cb072d014ce4febee610a7cf888bd7c10d508f582aa"} err="failed to get container status \"d9246068f333c1ef64192cb072d014ce4febee610a7cf888bd7c10d508f582aa\": rpc error: code = NotFound desc = could not find container \"d9246068f333c1ef64192cb072d014ce4febee610a7cf888bd7c10d508f582aa\": container with ID starting with d9246068f333c1ef64192cb072d014ce4febee610a7cf888bd7c10d508f582aa not found: ID does not exist"
Nov 02 02:06:58.000444 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:58.000409    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"]
Nov 02 02:06:58.005160 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:58.005129    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7a24b9-openshift-default-795b87c889-5k6kp"]
Nov 02 02:06:58.696588 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:06:58.696550    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c95abfc-9178-4ce3-9e78-cab75960a726" path="/var/lib/kubelet/pods/6c95abfc-9178-4ce3-9e78-cab75960a726/volumes"
Nov 02 02:07:06.439733 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:06.439699    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8"]
Nov 02 02:07:06.455788 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:06.455763    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:07:07.295687 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:07.295653    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck"]
Nov 02 02:07:11.320006 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:07:11.319919    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:07:11.441022 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:11.440969    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8" podUID="8425af9f-79bc-4f19-9109-f6712d7b8198" containerName="payload-processing" containerID="cri-o://52f34d071b5acd5cfda0b6611b43cf0c2bab1680f506a05e90eb6346b8f8ed3e" gracePeriod=25
Nov 02 02:07:11.721662 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:11.721638    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8"
Nov 02 02:07:11.821270 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:11.821234    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8425af9f-79bc-4f19-9109-f6712d7b8198-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8425af9f-79bc-4f19-9109-f6712d7b8198-plugins-config-volume\") pod \"8425af9f-79bc-4f19-9109-f6712d7b8198\" (UID: \"8425af9f-79bc-4f19-9109-f6712d7b8198\") "
Nov 02 02:07:11.821426 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:11.821363    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8425af9f-79bc-4f19-9109-f6712d7b8198-kube-api-access-lpf67\" (UniqueName: \"kubernetes.io/projected/8425af9f-79bc-4f19-9109-f6712d7b8198-kube-api-access-lpf67\") pod \"8425af9f-79bc-4f19-9109-f6712d7b8198\" (UID: \"8425af9f-79bc-4f19-9109-f6712d7b8198\") "
Nov 02 02:07:11.821560 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:11.821540    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8425af9f-79bc-4f19-9109-f6712d7b8198-plugins-config-volume" pod "8425af9f-79bc-4f19-9109-f6712d7b8198" (UID: "8425af9f-79bc-4f19-9109-f6712d7b8198"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:11.822048 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:11.822019    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8425af9f-79bc-4f19-9109-f6712d7b8198-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:11.823576 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:11.823558    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8425af9f-79bc-4f19-9109-f6712d7b8198-kube-api-access-lpf67" pod "8425af9f-79bc-4f19-9109-f6712d7b8198" (UID: "8425af9f-79bc-4f19-9109-f6712d7b8198"). InnerVolumeSpecName "kube-api-access-lpf67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:11.923472 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:11.923390    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpf67\" (UniqueName: \"kubernetes.io/projected/8425af9f-79bc-4f19-9109-f6712d7b8198-kube-api-access-lpf67\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:12.059396 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:12.059356    5875 generic.go:356] "Generic (PLEG): container finished" podID="8425af9f-79bc-4f19-9109-f6712d7b8198" containerID="52f34d071b5acd5cfda0b6611b43cf0c2bab1680f506a05e90eb6346b8f8ed3e" exitCode=0
Nov 02 02:07:12.059593 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:12.059403    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8"
Nov 02 02:07:12.059593 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:12.059440    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8" event={"ID":"8425af9f-79bc-4f19-9109-f6712d7b8198","Type":"ContainerDied","Data":"52f34d071b5acd5cfda0b6611b43cf0c2bab1680f506a05e90eb6346b8f8ed3e"}
Nov 02 02:07:12.059593 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:12.059512    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8" event={"ID":"8425af9f-79bc-4f19-9109-f6712d7b8198","Type":"ContainerDied","Data":"df264f18fe23dcaf0fd9fb0dffb45ce9fe8892849e8e6f58a23e2d3cd8a59c03"}
Nov 02 02:07:12.059593 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:12.059545    5875 scope.go:122] "RemoveContainer" containerID="52f34d071b5acd5cfda0b6611b43cf0c2bab1680f506a05e90eb6346b8f8ed3e"
Nov 02 02:07:12.070828 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:12.070809    5875 scope.go:122] "RemoveContainer" containerID="52f34d071b5acd5cfda0b6611b43cf0c2bab1680f506a05e90eb6346b8f8ed3e"
Nov 02 02:07:12.071149 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:07:12.071129    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f34d071b5acd5cfda0b6611b43cf0c2bab1680f506a05e90eb6346b8f8ed3e\": container with ID starting with 52f34d071b5acd5cfda0b6611b43cf0c2bab1680f506a05e90eb6346b8f8ed3e not found: ID does not exist" containerID="52f34d071b5acd5cfda0b6611b43cf0c2bab1680f506a05e90eb6346b8f8ed3e"
Nov 02 02:07:12.071209 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:12.071160    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f34d071b5acd5cfda0b6611b43cf0c2bab1680f506a05e90eb6346b8f8ed3e"} err="failed to get container status \"52f34d071b5acd5cfda0b6611b43cf0c2bab1680f506a05e90eb6346b8f8ed3e\": rpc error: code = NotFound desc = could not find container \"52f34d071b5acd5cfda0b6611b43cf0c2bab1680f506a05e90eb6346b8f8ed3e\": container with ID starting with 52f34d071b5acd5cfda0b6611b43cf0c2bab1680f506a05e90eb6346b8f8ed3e not found: ID does not exist"
Nov 02 02:07:12.224571 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:12.224544    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8"]
Nov 02 02:07:12.229805 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:12.229779    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-73e1cc-6d8d5d9695-kc7c8"]
Nov 02 02:07:12.297038 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:12.297004    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck" podUID="e52a668a-65c7-4961-9af7-8f176d9f2c2b" containerName="payload-processing" containerID="cri-o://7d6a1f1320e2d0575b2a84eeaca5c016624526584dc39ac494df74af04df3996" gracePeriod=25
Nov 02 02:07:12.568131 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:12.568101    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck"
Nov 02 02:07:12.633471 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:12.633443    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e52a668a-65c7-4961-9af7-8f176d9f2c2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52a668a-65c7-4961-9af7-8f176d9f2c2b-plugins-config-volume\") pod \"e52a668a-65c7-4961-9af7-8f176d9f2c2b\" (UID: \"e52a668a-65c7-4961-9af7-8f176d9f2c2b\") "
Nov 02 02:07:12.633662 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:12.633577    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52a668a-65c7-4961-9af7-8f176d9f2c2b-kube-api-access-64zvf\" (UniqueName: \"kubernetes.io/projected/e52a668a-65c7-4961-9af7-8f176d9f2c2b-kube-api-access-64zvf\") pod \"e52a668a-65c7-4961-9af7-8f176d9f2c2b\" (UID: \"e52a668a-65c7-4961-9af7-8f176d9f2c2b\") "
Nov 02 02:07:12.633848 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:12.633825    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e52a668a-65c7-4961-9af7-8f176d9f2c2b-plugins-config-volume" pod "e52a668a-65c7-4961-9af7-8f176d9f2c2b" (UID: "e52a668a-65c7-4961-9af7-8f176d9f2c2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:12.634130 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:12.634113    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52a668a-65c7-4961-9af7-8f176d9f2c2b-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:12.635778 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:12.635752    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52a668a-65c7-4961-9af7-8f176d9f2c2b-kube-api-access-64zvf" pod "e52a668a-65c7-4961-9af7-8f176d9f2c2b" (UID: "e52a668a-65c7-4961-9af7-8f176d9f2c2b"). InnerVolumeSpecName "kube-api-access-64zvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:12.699479 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:12.699450    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8425af9f-79bc-4f19-9109-f6712d7b8198" path="/var/lib/kubelet/pods/8425af9f-79bc-4f19-9109-f6712d7b8198/volumes"
Nov 02 02:07:12.736261 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:12.736213    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64zvf\" (UniqueName: \"kubernetes.io/projected/e52a668a-65c7-4961-9af7-8f176d9f2c2b-kube-api-access-64zvf\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:13.068080 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:13.068042    5875 generic.go:356] "Generic (PLEG): container finished" podID="e52a668a-65c7-4961-9af7-8f176d9f2c2b" containerID="7d6a1f1320e2d0575b2a84eeaca5c016624526584dc39ac494df74af04df3996" exitCode=0
Nov 02 02:07:13.068292 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:13.068090    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck"
Nov 02 02:07:13.068292 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:13.068129    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck" event={"ID":"e52a668a-65c7-4961-9af7-8f176d9f2c2b","Type":"ContainerDied","Data":"7d6a1f1320e2d0575b2a84eeaca5c016624526584dc39ac494df74af04df3996"}
Nov 02 02:07:13.068292 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:13.068182    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck" event={"ID":"e52a668a-65c7-4961-9af7-8f176d9f2c2b","Type":"ContainerDied","Data":"e91702ecebbfc894608a2e3ef6722216a36b3e84ba837a8baab4aa99a3175b07"}
Nov 02 02:07:13.068292 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:13.068214    5875 scope.go:122] "RemoveContainer" containerID="7d6a1f1320e2d0575b2a84eeaca5c016624526584dc39ac494df74af04df3996"
Nov 02 02:07:13.078634 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:13.078617    5875 scope.go:122] "RemoveContainer" containerID="7d6a1f1320e2d0575b2a84eeaca5c016624526584dc39ac494df74af04df3996"
Nov 02 02:07:13.078915 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:07:13.078887    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d6a1f1320e2d0575b2a84eeaca5c016624526584dc39ac494df74af04df3996\": container with ID starting with 7d6a1f1320e2d0575b2a84eeaca5c016624526584dc39ac494df74af04df3996 not found: ID does not exist" containerID="7d6a1f1320e2d0575b2a84eeaca5c016624526584dc39ac494df74af04df3996"
Nov 02 02:07:13.078966 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:13.078923    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d6a1f1320e2d0575b2a84eeaca5c016624526584dc39ac494df74af04df3996"} err="failed to get container status \"7d6a1f1320e2d0575b2a84eeaca5c016624526584dc39ac494df74af04df3996\": rpc error: code = NotFound desc = could not find container \"7d6a1f1320e2d0575b2a84eeaca5c016624526584dc39ac494df74af04df3996\": container with ID starting with 7d6a1f1320e2d0575b2a84eeaca5c016624526584dc39ac494df74af04df3996 not found: ID does not exist"
Nov 02 02:07:13.232064 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:13.232032    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck"]
Nov 02 02:07:13.237568 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:13.237542    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aa1597-85c7b9d4fc-tt5ck"]
Nov 02 02:07:14.696576 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:14.696547    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e52a668a-65c7-4961-9af7-8f176d9f2c2b" path="/var/lib/kubelet/pods/e52a668a-65c7-4961-9af7-8f176d9f2c2b/volumes"
Nov 02 02:07:18.629351 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:18.629321    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"]
Nov 02 02:07:18.629742 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:18.629517    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk" podUID="fda26851-8e30-489a-896d-73d9bcdd783f" containerName="istio-proxy" containerID="cri-o://67335c33dd2e519c62740b1864d34bd18dee90f4324a08cc1f480badaa0982a1" gracePeriod=30
Nov 02 02:07:23.027406 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:23.027369    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv"]
Nov 02 02:07:23.027820 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:23.027621    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv" podUID="fc421d13-d643-4058-bcfa-1a44a56365eb" containerName="maas-api" containerID="cri-o://eaf0c26077c94fecc149ebb30cf7d4dcb5a6aa075b59e97ec80bc7af6e9e9161" gracePeriod=30
Nov 02 02:07:23.146020 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:23.145992    5875 generic.go:356] "Generic (PLEG): container finished" podID="fc421d13-d643-4058-bcfa-1a44a56365eb" containerID="eaf0c26077c94fecc149ebb30cf7d4dcb5a6aa075b59e97ec80bc7af6e9e9161" exitCode=0
Nov 02 02:07:23.146207 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:23.146188    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv" event={"ID":"fc421d13-d643-4058-bcfa-1a44a56365eb","Type":"ContainerDied","Data":"eaf0c26077c94fecc149ebb30cf7d4dcb5a6aa075b59e97ec80bc7af6e9e9161"}
Nov 02 02:07:23.357568 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:23.357542    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv"
Nov 02 02:07:23.404914 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:23.404884    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc421d13-d643-4058-bcfa-1a44a56365eb-kube-api-access-xj5h9\" (UniqueName: \"kubernetes.io/projected/fc421d13-d643-4058-bcfa-1a44a56365eb-kube-api-access-xj5h9\") pod \"fc421d13-d643-4058-bcfa-1a44a56365eb\" (UID: \"fc421d13-d643-4058-bcfa-1a44a56365eb\") "
Nov 02 02:07:23.405063 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:23.404945    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc421d13-d643-4058-bcfa-1a44a56365eb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc421d13-d643-4058-bcfa-1a44a56365eb-metrics-certs\") pod \"fc421d13-d643-4058-bcfa-1a44a56365eb\" (UID: \"fc421d13-d643-4058-bcfa-1a44a56365eb\") "
Nov 02 02:07:23.405063 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:23.404971    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc421d13-d643-4058-bcfa-1a44a56365eb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc421d13-d643-4058-bcfa-1a44a56365eb-openshift-service-ca\") pod \"fc421d13-d643-4058-bcfa-1a44a56365eb\" (UID: \"fc421d13-d643-4058-bcfa-1a44a56365eb\") "
Nov 02 02:07:23.405063 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:23.404998    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc421d13-d643-4058-bcfa-1a44a56365eb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc421d13-d643-4058-bcfa-1a44a56365eb-maas-api-tls\") pod \"fc421d13-d643-4058-bcfa-1a44a56365eb\" (UID: \"fc421d13-d643-4058-bcfa-1a44a56365eb\") "
Nov 02 02:07:23.405408 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:23.405380    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc421d13-d643-4058-bcfa-1a44a56365eb-openshift-service-ca" pod "fc421d13-d643-4058-bcfa-1a44a56365eb" (UID: "fc421d13-d643-4058-bcfa-1a44a56365eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:23.405675 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:23.405658    5875 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc421d13-d643-4058-bcfa-1a44a56365eb-openshift-service-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:23.407350 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:23.407321    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc421d13-d643-4058-bcfa-1a44a56365eb-maas-api-tls" pod "fc421d13-d643-4058-bcfa-1a44a56365eb" (UID: "fc421d13-d643-4058-bcfa-1a44a56365eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:07:23.407350 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:23.407340    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc421d13-d643-4058-bcfa-1a44a56365eb-metrics-certs" pod "fc421d13-d643-4058-bcfa-1a44a56365eb" (UID: "fc421d13-d643-4058-bcfa-1a44a56365eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:07:23.407512 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:23.407496    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc421d13-d643-4058-bcfa-1a44a56365eb-kube-api-access-xj5h9" pod "fc421d13-d643-4058-bcfa-1a44a56365eb" (UID: "fc421d13-d643-4058-bcfa-1a44a56365eb"). InnerVolumeSpecName "kube-api-access-xj5h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:23.511532 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:23.511441    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj5h9\" (UniqueName: \"kubernetes.io/projected/fc421d13-d643-4058-bcfa-1a44a56365eb-kube-api-access-xj5h9\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:23.511532 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:23.511483    5875 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc421d13-d643-4058-bcfa-1a44a56365eb-metrics-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:23.511532 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:23.511501    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc421d13-d643-4058-bcfa-1a44a56365eb-maas-api-tls\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:24.160579 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.160325    5875 generic.go:356] "Generic (PLEG): container finished" podID="fda26851-8e30-489a-896d-73d9bcdd783f" containerID="67335c33dd2e519c62740b1864d34bd18dee90f4324a08cc1f480badaa0982a1" exitCode=0
Nov 02 02:07:24.160579 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.160370    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk" event={"ID":"fda26851-8e30-489a-896d-73d9bcdd783f","Type":"ContainerDied","Data":"67335c33dd2e519c62740b1864d34bd18dee90f4324a08cc1f480badaa0982a1"}
Nov 02 02:07:24.160579 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.160448    5875 scope.go:122] "RemoveContainer" containerID="4eff3eed579d2fc8030603ab4af2ef048a5dac16cfe3c875cdf4bf341d9b5d82"
Nov 02 02:07:24.163323 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.163293    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv" event={"ID":"fc421d13-d643-4058-bcfa-1a44a56365eb","Type":"ContainerDied","Data":"418ab2c33dd35907e4511e9b5ef9bdce910cccb59053d584feaa2a501e58b603"}
Nov 02 02:07:24.163447 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.163410    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv"
Nov 02 02:07:24.181353 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.181277    5875 scope.go:122] "RemoveContainer" containerID="eaf0c26077c94fecc149ebb30cf7d4dcb5a6aa075b59e97ec80bc7af6e9e9161"
Nov 02 02:07:24.330018 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.329946    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv"]
Nov 02 02:07:24.332514 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.332493    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:07:24.378753 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.378726    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-abc85a-5688dc6f6c-f48vv"]
Nov 02 02:07:24.492328 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.492293    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-workload-socket\") pod \"fda26851-8e30-489a-896d-73d9bcdd783f\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") "
Nov 02 02:07:24.492538 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.492408    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-istio-data\") pod \"fda26851-8e30-489a-896d-73d9bcdd783f\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") "
Nov 02 02:07:24.492538 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.492469    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fda26851-8e30-489a-896d-73d9bcdd783f-kube-api-access-knzhv\" (UniqueName: \"kubernetes.io/projected/fda26851-8e30-489a-896d-73d9bcdd783f-kube-api-access-knzhv\") pod \"fda26851-8e30-489a-896d-73d9bcdd783f\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") "
Nov 02 02:07:24.492673 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.492534    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-istio-envoy\") pod \"fda26851-8e30-489a-896d-73d9bcdd783f\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") "
Nov 02 02:07:24.492673 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.492577    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fda26851-8e30-489a-896d-73d9bcdd783f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fda26851-8e30-489a-896d-73d9bcdd783f-istio-podinfo\") pod \"fda26851-8e30-489a-896d-73d9bcdd783f\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") "
Nov 02 02:07:24.492673 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.492584    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-workload-socket" pod "fda26851-8e30-489a-896d-73d9bcdd783f" (UID: "fda26851-8e30-489a-896d-73d9bcdd783f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:24.492673 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.492601    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-workload-certs\") pod \"fda26851-8e30-489a-896d-73d9bcdd783f\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") "
Nov 02 02:07:24.492911 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.492677    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fda26851-8e30-489a-896d-73d9bcdd783f-istio-token\" (UniqueName: \"kubernetes.io/projected/fda26851-8e30-489a-896d-73d9bcdd783f-istio-token\") pod \"fda26851-8e30-489a-896d-73d9bcdd783f\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") "
Nov 02 02:07:24.492911 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.492728    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-credential-socket\") pod \"fda26851-8e30-489a-896d-73d9bcdd783f\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") "
Nov 02 02:07:24.492911 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.492805    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fda26851-8e30-489a-896d-73d9bcdd783f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fda26851-8e30-489a-896d-73d9bcdd783f-istiod-ca-cert\") pod \"fda26851-8e30-489a-896d-73d9bcdd783f\" (UID: \"fda26851-8e30-489a-896d-73d9bcdd783f\") "
Nov 02 02:07:24.493568 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.493528    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-credential-socket" pod "fda26851-8e30-489a-896d-73d9bcdd783f" (UID: "fda26851-8e30-489a-896d-73d9bcdd783f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:24.493568 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.493544    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:24.493783 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.493564    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-workload-certs" pod "fda26851-8e30-489a-896d-73d9bcdd783f" (UID: "fda26851-8e30-489a-896d-73d9bcdd783f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:24.493783 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.493694    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-istio-data" pod "fda26851-8e30-489a-896d-73d9bcdd783f" (UID: "fda26851-8e30-489a-896d-73d9bcdd783f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:24.493901 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.493802    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fda26851-8e30-489a-896d-73d9bcdd783f-istiod-ca-cert" pod "fda26851-8e30-489a-896d-73d9bcdd783f" (UID: "fda26851-8e30-489a-896d-73d9bcdd783f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:24.495571 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.495539    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda26851-8e30-489a-896d-73d9bcdd783f-kube-api-access-knzhv" pod "fda26851-8e30-489a-896d-73d9bcdd783f" (UID: "fda26851-8e30-489a-896d-73d9bcdd783f"). InnerVolumeSpecName "kube-api-access-knzhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:24.495821 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.495797    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-istio-envoy" pod "fda26851-8e30-489a-896d-73d9bcdd783f" (UID: "fda26851-8e30-489a-896d-73d9bcdd783f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:24.495929 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.495913    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fda26851-8e30-489a-896d-73d9bcdd783f-istio-podinfo" pod "fda26851-8e30-489a-896d-73d9bcdd783f" (UID: "fda26851-8e30-489a-896d-73d9bcdd783f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:07:24.496210 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.496196    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda26851-8e30-489a-896d-73d9bcdd783f-istio-token" pod "fda26851-8e30-489a-896d-73d9bcdd783f" (UID: "fda26851-8e30-489a-896d-73d9bcdd783f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:24.594761 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.594691    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:24.594761 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.594720    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fda26851-8e30-489a-896d-73d9bcdd783f-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:24.594761 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.594730    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:24.594761 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.594739    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fda26851-8e30-489a-896d-73d9bcdd783f-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:24.594761 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.594750    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:24.594761 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.594759    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fda26851-8e30-489a-896d-73d9bcdd783f-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:24.594761 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.594767    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fda26851-8e30-489a-896d-73d9bcdd783f-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:24.595110 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.594777    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knzhv\" (UniqueName: \"kubernetes.io/projected/fda26851-8e30-489a-896d-73d9bcdd783f-kube-api-access-knzhv\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:24.698264 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:24.698206    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc421d13-d643-4058-bcfa-1a44a56365eb" path="/var/lib/kubelet/pods/fc421d13-d643-4058-bcfa-1a44a56365eb/volumes"
Nov 02 02:07:25.188048 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:25.188024    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"
Nov 02 02:07:25.188547 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:25.188019    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk" event={"ID":"fda26851-8e30-489a-896d-73d9bcdd783f","Type":"ContainerDied","Data":"ad675c0ec177379d8cafcda3cd81466e0634870b775feb075307a13998036b35"}
Nov 02 02:07:25.188547 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:25.188149    5875 scope.go:122] "RemoveContainer" containerID="67335c33dd2e519c62740b1864d34bd18dee90f4324a08cc1f480badaa0982a1"
Nov 02 02:07:25.402636 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:25.402599    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"]
Nov 02 02:07:25.452092 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:25.452012    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-aa1597-openshift-default-796d69645f-4xwsk"]
Nov 02 02:07:26.695827 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:26.695785    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fda26851-8e30-489a-896d-73d9bcdd783f" path="/var/lib/kubelet/pods/fda26851-8e30-489a-896d-73d9bcdd783f/volumes"
Nov 02 02:07:29.477775 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:29.477744    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm"]
Nov 02 02:07:29.478300 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:29.477997    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm" podUID="3986fc99-87a5-41cf-bb18-0815f5e56d96" containerName="maas-api" containerID="cri-o://19dd10fa0ddfbec1100b1affe90d4ccb0cb7a486dc125ab31e066a27aa4b304b" gracePeriod=30
Nov 02 02:07:29.814846 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:29.814820    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm"
Nov 02 02:07:29.876514 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:29.876478    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3986fc99-87a5-41cf-bb18-0815f5e56d96-kube-api-access-pzv8x\" (UniqueName: \"kubernetes.io/projected/3986fc99-87a5-41cf-bb18-0815f5e56d96-kube-api-access-pzv8x\") pod \"3986fc99-87a5-41cf-bb18-0815f5e56d96\" (UID: \"3986fc99-87a5-41cf-bb18-0815f5e56d96\") "
Nov 02 02:07:29.876718 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:29.876605    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3986fc99-87a5-41cf-bb18-0815f5e56d96-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3986fc99-87a5-41cf-bb18-0815f5e56d96-metrics-certs\") pod \"3986fc99-87a5-41cf-bb18-0815f5e56d96\" (UID: \"3986fc99-87a5-41cf-bb18-0815f5e56d96\") "
Nov 02 02:07:29.876832 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:29.876753    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3986fc99-87a5-41cf-bb18-0815f5e56d96-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3986fc99-87a5-41cf-bb18-0815f5e56d96-maas-api-tls\") pod \"3986fc99-87a5-41cf-bb18-0815f5e56d96\" (UID: \"3986fc99-87a5-41cf-bb18-0815f5e56d96\") "
Nov 02 02:07:29.876832 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:29.876818    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3986fc99-87a5-41cf-bb18-0815f5e56d96-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3986fc99-87a5-41cf-bb18-0815f5e56d96-openshift-service-ca\") pod \"3986fc99-87a5-41cf-bb18-0815f5e56d96\" (UID: \"3986fc99-87a5-41cf-bb18-0815f5e56d96\") "
Nov 02 02:07:29.877572 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:29.877537    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3986fc99-87a5-41cf-bb18-0815f5e56d96-openshift-service-ca" pod "3986fc99-87a5-41cf-bb18-0815f5e56d96" (UID: "3986fc99-87a5-41cf-bb18-0815f5e56d96"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:29.879298 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:29.879274    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3986fc99-87a5-41cf-bb18-0815f5e56d96-metrics-certs" pod "3986fc99-87a5-41cf-bb18-0815f5e56d96" (UID: "3986fc99-87a5-41cf-bb18-0815f5e56d96"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:07:29.879538 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:29.879519    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3986fc99-87a5-41cf-bb18-0815f5e56d96-kube-api-access-pzv8x" pod "3986fc99-87a5-41cf-bb18-0815f5e56d96" (UID: "3986fc99-87a5-41cf-bb18-0815f5e56d96"). InnerVolumeSpecName "kube-api-access-pzv8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:29.881142 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:29.881118    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3986fc99-87a5-41cf-bb18-0815f5e56d96-maas-api-tls" pod "3986fc99-87a5-41cf-bb18-0815f5e56d96" (UID: "3986fc99-87a5-41cf-bb18-0815f5e56d96"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:07:29.978380 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:29.978353    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzv8x\" (UniqueName: \"kubernetes.io/projected/3986fc99-87a5-41cf-bb18-0815f5e56d96-kube-api-access-pzv8x\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:29.978380 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:29.978378    5875 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3986fc99-87a5-41cf-bb18-0815f5e56d96-metrics-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:29.978380 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:29.978387    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3986fc99-87a5-41cf-bb18-0815f5e56d96-maas-api-tls\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:29.978615 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:29.978396    5875 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3986fc99-87a5-41cf-bb18-0815f5e56d96-openshift-service-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:30.227204 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:30.227168    5875 generic.go:356] "Generic (PLEG): container finished" podID="3986fc99-87a5-41cf-bb18-0815f5e56d96" containerID="19dd10fa0ddfbec1100b1affe90d4ccb0cb7a486dc125ab31e066a27aa4b304b" exitCode=0
Nov 02 02:07:30.227397 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:30.227211    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm"
Nov 02 02:07:30.227397 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:30.227258    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm" event={"ID":"3986fc99-87a5-41cf-bb18-0815f5e56d96","Type":"ContainerDied","Data":"19dd10fa0ddfbec1100b1affe90d4ccb0cb7a486dc125ab31e066a27aa4b304b"}
Nov 02 02:07:30.227397 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:30.227311    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm" event={"ID":"3986fc99-87a5-41cf-bb18-0815f5e56d96","Type":"ContainerDied","Data":"90b36da7466df29cc2ff1b21ccab247d27e082637e4d680d1398dab7d54978cf"}
Nov 02 02:07:30.227397 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:30.227335    5875 scope.go:122] "RemoveContainer" containerID="19dd10fa0ddfbec1100b1affe90d4ccb0cb7a486dc125ab31e066a27aa4b304b"
Nov 02 02:07:30.238756 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:30.238738    5875 scope.go:122] "RemoveContainer" containerID="19dd10fa0ddfbec1100b1affe90d4ccb0cb7a486dc125ab31e066a27aa4b304b"
Nov 02 02:07:30.239035 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:07:30.239014    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19dd10fa0ddfbec1100b1affe90d4ccb0cb7a486dc125ab31e066a27aa4b304b\": container with ID starting with 19dd10fa0ddfbec1100b1affe90d4ccb0cb7a486dc125ab31e066a27aa4b304b not found: ID does not exist" containerID="19dd10fa0ddfbec1100b1affe90d4ccb0cb7a486dc125ab31e066a27aa4b304b"
Nov 02 02:07:30.239092 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:30.239042    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19dd10fa0ddfbec1100b1affe90d4ccb0cb7a486dc125ab31e066a27aa4b304b"} err="failed to get container status \"19dd10fa0ddfbec1100b1affe90d4ccb0cb7a486dc125ab31e066a27aa4b304b\": rpc error: code = NotFound desc = could not find container \"19dd10fa0ddfbec1100b1affe90d4ccb0cb7a486dc125ab31e066a27aa4b304b\": container with ID starting with 19dd10fa0ddfbec1100b1affe90d4ccb0cb7a486dc125ab31e066a27aa4b304b not found: ID does not exist"
Nov 02 02:07:30.446260 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:30.443323    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm"]
Nov 02 02:07:30.465173 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:30.464349    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-73e1cc-55b65778ff-lcjfm"]
Nov 02 02:07:30.660891 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:30.660808    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr"]
Nov 02 02:07:30.661318 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:30.661029    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr" podUID="314d6835-e1c6-48ba-b4eb-a744a699c061" containerName="maas-api" containerID="cri-o://383100b6d34c35b3ca81bea70223dada3904205ca988169e5ffe6604b6e1dc25" gracePeriod=30
Nov 02 02:07:30.701588 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:30.701552    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3986fc99-87a5-41cf-bb18-0815f5e56d96" path="/var/lib/kubelet/pods/3986fc99-87a5-41cf-bb18-0815f5e56d96/volumes"
Nov 02 02:07:30.973059 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:30.972938    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr"
Nov 02 02:07:31.042950 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:31.042916    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/314d6835-e1c6-48ba-b4eb-a744a699c061-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/314d6835-e1c6-48ba-b4eb-a744a699c061-maas-api-tls\") pod \"314d6835-e1c6-48ba-b4eb-a744a699c061\" (UID: \"314d6835-e1c6-48ba-b4eb-a744a699c061\") "
Nov 02 02:07:31.043132 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:31.042973    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/314d6835-e1c6-48ba-b4eb-a744a699c061-metrics-certs\" (UniqueName: \"kubernetes.io/secret/314d6835-e1c6-48ba-b4eb-a744a699c061-metrics-certs\") pod \"314d6835-e1c6-48ba-b4eb-a744a699c061\" (UID: \"314d6835-e1c6-48ba-b4eb-a744a699c061\") "
Nov 02 02:07:31.043132 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:31.043035    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/314d6835-e1c6-48ba-b4eb-a744a699c061-kube-api-access-lhr6v\" (UniqueName: \"kubernetes.io/projected/314d6835-e1c6-48ba-b4eb-a744a699c061-kube-api-access-lhr6v\") pod \"314d6835-e1c6-48ba-b4eb-a744a699c061\" (UID: \"314d6835-e1c6-48ba-b4eb-a744a699c061\") "
Nov 02 02:07:31.044343 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:31.043341    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/314d6835-e1c6-48ba-b4eb-a744a699c061-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/314d6835-e1c6-48ba-b4eb-a744a699c061-openshift-service-ca\") pod \"314d6835-e1c6-48ba-b4eb-a744a699c061\" (UID: \"314d6835-e1c6-48ba-b4eb-a744a699c061\") "
Nov 02 02:07:31.044697 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:31.044663    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/314d6835-e1c6-48ba-b4eb-a744a699c061-openshift-service-ca" pod "314d6835-e1c6-48ba-b4eb-a744a699c061" (UID: "314d6835-e1c6-48ba-b4eb-a744a699c061"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:31.048379 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:31.048340    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/314d6835-e1c6-48ba-b4eb-a744a699c061-metrics-certs" pod "314d6835-e1c6-48ba-b4eb-a744a699c061" (UID: "314d6835-e1c6-48ba-b4eb-a744a699c061"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:07:31.055475 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:31.055054    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/314d6835-e1c6-48ba-b4eb-a744a699c061-kube-api-access-lhr6v" pod "314d6835-e1c6-48ba-b4eb-a744a699c061" (UID: "314d6835-e1c6-48ba-b4eb-a744a699c061"). InnerVolumeSpecName "kube-api-access-lhr6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:31.056192 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:31.056161    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/314d6835-e1c6-48ba-b4eb-a744a699c061-maas-api-tls" pod "314d6835-e1c6-48ba-b4eb-a744a699c061" (UID: "314d6835-e1c6-48ba-b4eb-a744a699c061"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:07:31.146032 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:31.145991    5875 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/314d6835-e1c6-48ba-b4eb-a744a699c061-openshift-service-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:31.146032 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:31.146024    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/314d6835-e1c6-48ba-b4eb-a744a699c061-maas-api-tls\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:31.146032 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:31.146038    5875 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/314d6835-e1c6-48ba-b4eb-a744a699c061-metrics-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:31.146272 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:31.146050    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhr6v\" (UniqueName: \"kubernetes.io/projected/314d6835-e1c6-48ba-b4eb-a744a699c061-kube-api-access-lhr6v\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:31.237543 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:31.237508    5875 generic.go:356] "Generic (PLEG): container finished" podID="314d6835-e1c6-48ba-b4eb-a744a699c061" containerID="383100b6d34c35b3ca81bea70223dada3904205ca988169e5ffe6604b6e1dc25" exitCode=0
Nov 02 02:07:31.237693 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:31.237576    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr"
Nov 02 02:07:31.237693 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:31.237593    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr" event={"ID":"314d6835-e1c6-48ba-b4eb-a744a699c061","Type":"ContainerDied","Data":"383100b6d34c35b3ca81bea70223dada3904205ca988169e5ffe6604b6e1dc25"}
Nov 02 02:07:31.237693 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:31.237671    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr" event={"ID":"314d6835-e1c6-48ba-b4eb-a744a699c061","Type":"ContainerDied","Data":"5075a892a26d5365ab61494d813f6a5d1c8880eb3c7273d65e0d9d4b3403ca88"}
Nov 02 02:07:31.237693 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:31.237693    5875 scope.go:122] "RemoveContainer" containerID="383100b6d34c35b3ca81bea70223dada3904205ca988169e5ffe6604b6e1dc25"
Nov 02 02:07:31.248882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:31.248865    5875 scope.go:122] "RemoveContainer" containerID="383100b6d34c35b3ca81bea70223dada3904205ca988169e5ffe6604b6e1dc25"
Nov 02 02:07:31.249157 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:07:31.249139    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"383100b6d34c35b3ca81bea70223dada3904205ca988169e5ffe6604b6e1dc25\": container with ID starting with 383100b6d34c35b3ca81bea70223dada3904205ca988169e5ffe6604b6e1dc25 not found: ID does not exist" containerID="383100b6d34c35b3ca81bea70223dada3904205ca988169e5ffe6604b6e1dc25"
Nov 02 02:07:31.249207 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:31.249166    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"383100b6d34c35b3ca81bea70223dada3904205ca988169e5ffe6604b6e1dc25"} err="failed to get container status \"383100b6d34c35b3ca81bea70223dada3904205ca988169e5ffe6604b6e1dc25\": rpc error: code = NotFound desc = could not find container \"383100b6d34c35b3ca81bea70223dada3904205ca988169e5ffe6604b6e1dc25\": container with ID starting with 383100b6d34c35b3ca81bea70223dada3904205ca988169e5ffe6604b6e1dc25 not found: ID does not exist"
Nov 02 02:07:31.430170 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:31.430126    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr"]
Nov 02 02:07:31.435193 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:31.435167    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-129de4-77bc59fd7-gcxrr"]
Nov 02 02:07:32.739095 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:32.739064    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="314d6835-e1c6-48ba-b4eb-a744a699c061" path="/var/lib/kubelet/pods/314d6835-e1c6-48ba-b4eb-a744a699c061/volumes"
Nov 02 02:07:32.980662 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:32.980625    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"]
Nov 02 02:07:32.980851 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:32.980817    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592" podUID="eb67a90f-51c6-4440-923d-db0edea22576" containerName="istio-proxy" containerID="cri-o://fda56a66cfddb674c9b940aac1ffd002b969a38238262737bbd37383a1ee7b48" gracePeriod=30
Nov 02 02:07:34.757455 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:34.757424    5875 patch_prober.go:29] interesting pod/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:07:34.757916 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:34.757483    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592" podUID="eb67a90f-51c6-4440-923d-db0edea22576" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:07:34.979856 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:34.979815    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf"]
Nov 02 02:07:38.265490 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.265424    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:07:38.357183 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.357153    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb67a90f-51c6-4440-923d-db0edea22576-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb67a90f-51c6-4440-923d-db0edea22576-istio-podinfo\") pod \"eb67a90f-51c6-4440-923d-db0edea22576\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") "
Nov 02 02:07:38.357379 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.357202    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-workload-certs\") pod \"eb67a90f-51c6-4440-923d-db0edea22576\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") "
Nov 02 02:07:38.357379 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.357245    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-credential-socket\") pod \"eb67a90f-51c6-4440-923d-db0edea22576\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") "
Nov 02 02:07:38.357379 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.357268    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-istio-envoy\") pod \"eb67a90f-51c6-4440-923d-db0edea22576\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") "
Nov 02 02:07:38.357379 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.357359    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-istio-data\") pod \"eb67a90f-51c6-4440-923d-db0edea22576\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") "
Nov 02 02:07:38.357608 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.357402    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-workload-socket\") pod \"eb67a90f-51c6-4440-923d-db0edea22576\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") "
Nov 02 02:07:38.357608 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.357446    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb67a90f-51c6-4440-923d-db0edea22576-kube-api-access-mqbcx\" (UniqueName: \"kubernetes.io/projected/eb67a90f-51c6-4440-923d-db0edea22576-kube-api-access-mqbcx\") pod \"eb67a90f-51c6-4440-923d-db0edea22576\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") "
Nov 02 02:07:38.357608 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.357520    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb67a90f-51c6-4440-923d-db0edea22576-istio-token\" (UniqueName: \"kubernetes.io/projected/eb67a90f-51c6-4440-923d-db0edea22576-istio-token\") pod \"eb67a90f-51c6-4440-923d-db0edea22576\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") "
Nov 02 02:07:38.357766 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.357619    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-workload-certs" pod "eb67a90f-51c6-4440-923d-db0edea22576" (UID: "eb67a90f-51c6-4440-923d-db0edea22576"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:38.357766 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.357711    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-istio-data" pod "eb67a90f-51c6-4440-923d-db0edea22576" (UID: "eb67a90f-51c6-4440-923d-db0edea22576"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:38.357766 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.357740    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-workload-socket" pod "eb67a90f-51c6-4440-923d-db0edea22576" (UID: "eb67a90f-51c6-4440-923d-db0edea22576"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:38.357935 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.357782    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb67a90f-51c6-4440-923d-db0edea22576-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb67a90f-51c6-4440-923d-db0edea22576-istiod-ca-cert\") pod \"eb67a90f-51c6-4440-923d-db0edea22576\" (UID: \"eb67a90f-51c6-4440-923d-db0edea22576\") "
Nov 02 02:07:38.357935 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.357792    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-credential-socket" pod "eb67a90f-51c6-4440-923d-db0edea22576" (UID: "eb67a90f-51c6-4440-923d-db0edea22576"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:38.358160 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.358132    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb67a90f-51c6-4440-923d-db0edea22576-istiod-ca-cert" pod "eb67a90f-51c6-4440-923d-db0edea22576" (UID: "eb67a90f-51c6-4440-923d-db0edea22576"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:38.358605 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.358526    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb67a90f-51c6-4440-923d-db0edea22576-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:38.358605 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.358553    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:38.358605 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.358569    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:38.358605 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.358584    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:38.358605 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.358600    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:38.359983 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.359950    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb67a90f-51c6-4440-923d-db0edea22576-istio-podinfo" pod "eb67a90f-51c6-4440-923d-db0edea22576" (UID: "eb67a90f-51c6-4440-923d-db0edea22576"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:07:38.360155 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.360131    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb67a90f-51c6-4440-923d-db0edea22576-kube-api-access-mqbcx" pod "eb67a90f-51c6-4440-923d-db0edea22576" (UID: "eb67a90f-51c6-4440-923d-db0edea22576"). InnerVolumeSpecName "kube-api-access-mqbcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:38.360293 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.360275    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb67a90f-51c6-4440-923d-db0edea22576-istio-token" pod "eb67a90f-51c6-4440-923d-db0edea22576" (UID: "eb67a90f-51c6-4440-923d-db0edea22576"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:38.360346 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.360325    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-istio-envoy" pod "eb67a90f-51c6-4440-923d-db0edea22576" (UID: "eb67a90f-51c6-4440-923d-db0edea22576"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:38.395516 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.395488    5875 generic.go:356] "Generic (PLEG): container finished" podID="eb67a90f-51c6-4440-923d-db0edea22576" containerID="fda56a66cfddb674c9b940aac1ffd002b969a38238262737bbd37383a1ee7b48" exitCode=0
Nov 02 02:07:38.395632 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.395541    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"
Nov 02 02:07:38.395632 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.395563    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592" event={"ID":"eb67a90f-51c6-4440-923d-db0edea22576","Type":"ContainerDied","Data":"fda56a66cfddb674c9b940aac1ffd002b969a38238262737bbd37383a1ee7b48"}
Nov 02 02:07:38.395632 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.395617    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592" event={"ID":"eb67a90f-51c6-4440-923d-db0edea22576","Type":"ContainerDied","Data":"9af59f28cbb75cb65b522816e0e8388988c093c0daf95d4bde76ec16c4dc06f9"}
Nov 02 02:07:38.395763 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.395641    5875 scope.go:122] "RemoveContainer" containerID="fda56a66cfddb674c9b940aac1ffd002b969a38238262737bbd37383a1ee7b48"
Nov 02 02:07:38.407102 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.407082    5875 scope.go:122] "RemoveContainer" containerID="218cfc7752e8ecb3b44a5d1566a3b02e44e0596529b3793861d7ecf021601e32"
Nov 02 02:07:38.416498 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.416480    5875 scope.go:122] "RemoveContainer" containerID="fda56a66cfddb674c9b940aac1ffd002b969a38238262737bbd37383a1ee7b48"
Nov 02 02:07:38.416748 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:07:38.416726    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fda56a66cfddb674c9b940aac1ffd002b969a38238262737bbd37383a1ee7b48\": container with ID starting with fda56a66cfddb674c9b940aac1ffd002b969a38238262737bbd37383a1ee7b48 not found: ID does not exist" containerID="fda56a66cfddb674c9b940aac1ffd002b969a38238262737bbd37383a1ee7b48"
Nov 02 02:07:38.416804 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.416755    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fda56a66cfddb674c9b940aac1ffd002b969a38238262737bbd37383a1ee7b48"} err="failed to get container status \"fda56a66cfddb674c9b940aac1ffd002b969a38238262737bbd37383a1ee7b48\": rpc error: code = NotFound desc = could not find container \"fda56a66cfddb674c9b940aac1ffd002b969a38238262737bbd37383a1ee7b48\": container with ID starting with fda56a66cfddb674c9b940aac1ffd002b969a38238262737bbd37383a1ee7b48 not found: ID does not exist"
Nov 02 02:07:38.416804 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.416773    5875 scope.go:122] "RemoveContainer" containerID="218cfc7752e8ecb3b44a5d1566a3b02e44e0596529b3793861d7ecf021601e32"
Nov 02 02:07:38.417015 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:07:38.416997    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"218cfc7752e8ecb3b44a5d1566a3b02e44e0596529b3793861d7ecf021601e32\": container with ID starting with 218cfc7752e8ecb3b44a5d1566a3b02e44e0596529b3793861d7ecf021601e32 not found: ID does not exist" containerID="218cfc7752e8ecb3b44a5d1566a3b02e44e0596529b3793861d7ecf021601e32"
Nov 02 02:07:38.417057 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.417022    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"218cfc7752e8ecb3b44a5d1566a3b02e44e0596529b3793861d7ecf021601e32"} err="failed to get container status \"218cfc7752e8ecb3b44a5d1566a3b02e44e0596529b3793861d7ecf021601e32\": rpc error: code = NotFound desc = could not find container \"218cfc7752e8ecb3b44a5d1566a3b02e44e0596529b3793861d7ecf021601e32\": container with ID starting with 218cfc7752e8ecb3b44a5d1566a3b02e44e0596529b3793861d7ecf021601e32 not found: ID does not exist"
Nov 02 02:07:38.460527 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.460496    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb67a90f-51c6-4440-923d-db0edea22576-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:38.460527 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.460524    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb67a90f-51c6-4440-923d-db0edea22576-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:38.460706 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.460539    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqbcx\" (UniqueName: \"kubernetes.io/projected/eb67a90f-51c6-4440-923d-db0edea22576-kube-api-access-mqbcx\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:38.460706 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.460552    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb67a90f-51c6-4440-923d-db0edea22576-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:38.549666 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.549592    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"]
Nov 02 02:07:38.555010 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.554986    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-abc85a-openshift-default-59867d7bbf-hn592"]
Nov 02 02:07:38.577893 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.577861    5875 scope.go:122] "RemoveContainer" containerID="f2d9690b23888988760664c3ad7ac8b0d867a1229a0502d44de05069bb6ff44c"
Nov 02 02:07:38.695887 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:38.695860    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb67a90f-51c6-4440-923d-db0edea22576" path="/var/lib/kubelet/pods/eb67a90f-51c6-4440-923d-db0edea22576/volumes"
Nov 02 02:07:39.567987 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:39.567952    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"]
Nov 02 02:07:39.568385 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:39.568238    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf" podUID="d4b7a180-01b7-4c32-93be-dcb96b6b3b42" containerName="istio-proxy" containerID="cri-o://f0a92bdc72fd824df97fc3528a7b6276d955dc32e5d8f2db4f91f239bdb8ba46" gracePeriod=30
Nov 02 02:07:39.982764 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:39.982716    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf" podUID="72076bb7-2227-495a-a707-5bfba8979668" containerName="payload-pre-processing" containerID="cri-o://13ee2e1b3a690d5404a86eaeb3124181212a4b395330556bf1f7d49e89a0f75a" gracePeriod=25
Nov 02 02:07:40.273164 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:40.273130    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf"
Nov 02 02:07:40.281752 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:40.281723    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72076bb7-2227-495a-a707-5bfba8979668-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72076bb7-2227-495a-a707-5bfba8979668-plugins-config-volume\") pod \"72076bb7-2227-495a-a707-5bfba8979668\" (UID: \"72076bb7-2227-495a-a707-5bfba8979668\") "
Nov 02 02:07:40.281955 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:40.281838    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72076bb7-2227-495a-a707-5bfba8979668-kube-api-access-bvjxs\" (UniqueName: \"kubernetes.io/projected/72076bb7-2227-495a-a707-5bfba8979668-kube-api-access-bvjxs\") pod \"72076bb7-2227-495a-a707-5bfba8979668\" (UID: \"72076bb7-2227-495a-a707-5bfba8979668\") "
Nov 02 02:07:40.282133 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:40.282110    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72076bb7-2227-495a-a707-5bfba8979668-plugins-config-volume" pod "72076bb7-2227-495a-a707-5bfba8979668" (UID: "72076bb7-2227-495a-a707-5bfba8979668"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:40.282498 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:40.282475    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72076bb7-2227-495a-a707-5bfba8979668-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:40.284634 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:40.284562    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72076bb7-2227-495a-a707-5bfba8979668-kube-api-access-bvjxs" pod "72076bb7-2227-495a-a707-5bfba8979668" (UID: "72076bb7-2227-495a-a707-5bfba8979668"). InnerVolumeSpecName "kube-api-access-bvjxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:40.377635 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:40.377597    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"]
Nov 02 02:07:40.384505 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:40.384474    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvjxs\" (UniqueName: \"kubernetes.io/projected/72076bb7-2227-495a-a707-5bfba8979668-kube-api-access-bvjxs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:40.412752 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:40.412715    5875 generic.go:356] "Generic (PLEG): container finished" podID="72076bb7-2227-495a-a707-5bfba8979668" containerID="13ee2e1b3a690d5404a86eaeb3124181212a4b395330556bf1f7d49e89a0f75a" exitCode=0
Nov 02 02:07:40.412940 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:40.412772    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf"
Nov 02 02:07:40.412940 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:40.412801    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf" event={"ID":"72076bb7-2227-495a-a707-5bfba8979668","Type":"ContainerDied","Data":"13ee2e1b3a690d5404a86eaeb3124181212a4b395330556bf1f7d49e89a0f75a"}
Nov 02 02:07:40.412940 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:40.412865    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf" event={"ID":"72076bb7-2227-495a-a707-5bfba8979668","Type":"ContainerDied","Data":"26c01d1e33ca082720be2801e61ce928466d4bbaf3fda8d992fab321202825d7"}
Nov 02 02:07:40.412940 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:40.412895    5875 scope.go:122] "RemoveContainer" containerID="13ee2e1b3a690d5404a86eaeb3124181212a4b395330556bf1f7d49e89a0f75a"
Nov 02 02:07:40.423970 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:40.423952    5875 scope.go:122] "RemoveContainer" containerID="13ee2e1b3a690d5404a86eaeb3124181212a4b395330556bf1f7d49e89a0f75a"
Nov 02 02:07:40.424281 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:07:40.424260    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ee2e1b3a690d5404a86eaeb3124181212a4b395330556bf1f7d49e89a0f75a\": container with ID starting with 13ee2e1b3a690d5404a86eaeb3124181212a4b395330556bf1f7d49e89a0f75a not found: ID does not exist" containerID="13ee2e1b3a690d5404a86eaeb3124181212a4b395330556bf1f7d49e89a0f75a"
Nov 02 02:07:40.424370 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:40.424289    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ee2e1b3a690d5404a86eaeb3124181212a4b395330556bf1f7d49e89a0f75a"} err="failed to get container status \"13ee2e1b3a690d5404a86eaeb3124181212a4b395330556bf1f7d49e89a0f75a\": rpc error: code = NotFound desc = could not find container \"13ee2e1b3a690d5404a86eaeb3124181212a4b395330556bf1f7d49e89a0f75a\": container with ID starting with 13ee2e1b3a690d5404a86eaeb3124181212a4b395330556bf1f7d49e89a0f75a not found: ID does not exist"
Nov 02 02:07:40.617970 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:40.617878    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf"]
Nov 02 02:07:40.623062 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:40.623038    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-606ca0-7d6b489687-pvdrf"]
Nov 02 02:07:40.695655 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:40.695623    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72076bb7-2227-495a-a707-5bfba8979668" path="/var/lib/kubelet/pods/72076bb7-2227-495a-a707-5bfba8979668/volumes"
Nov 02 02:07:41.101913 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:41.101880    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"]
Nov 02 02:07:41.346312 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:07:41.346277    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:07:41.420614 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:41.420530    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t" podUID="a0c5779f-1d48-4749-bcb0-8313b202142d" containerName="istio-proxy" containerID="cri-o://489d1950ec1c74f0377e702f56c7b8ab402593a229a78f42cc4b4c783bcdf856" gracePeriod=30
Nov 02 02:07:44.861396 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:44.861371    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 02:07:44.947915 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:44.947876    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istiod-ca-cert\") pod \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") "
Nov 02 02:07:44.948057 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:44.947923    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-kube-api-access-xgcd4\" (UniqueName: \"kubernetes.io/projected/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-kube-api-access-xgcd4\") pod \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") "
Nov 02 02:07:44.948057 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:44.948011    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-data\") pod \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") "
Nov 02 02:07:44.948057 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:44.948049    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-podinfo\") pod \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") "
Nov 02 02:07:44.948237 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:44.948090    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-workload-certs\") pod \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") "
Nov 02 02:07:44.948237 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:44.948122    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-token\" (UniqueName: \"kubernetes.io/projected/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-token\") pod \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") "
Nov 02 02:07:44.948237 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:44.948205    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-credential-socket\") pod \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") "
Nov 02 02:07:44.948366 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:44.948286    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-envoy\") pod \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") "
Nov 02 02:07:44.948366 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:44.948299    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istiod-ca-cert" pod "d4b7a180-01b7-4c32-93be-dcb96b6b3b42" (UID: "d4b7a180-01b7-4c32-93be-dcb96b6b3b42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:44.948366 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:44.948333    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-workload-socket\") pod \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\" (UID: \"d4b7a180-01b7-4c32-93be-dcb96b6b3b42\") "
Nov 02 02:07:44.948611 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:44.948590    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-workload-socket" pod "d4b7a180-01b7-4c32-93be-dcb96b6b3b42" (UID: "d4b7a180-01b7-4c32-93be-dcb96b6b3b42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:44.948837 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:44.948783    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-workload-certs" pod "d4b7a180-01b7-4c32-93be-dcb96b6b3b42" (UID: "d4b7a180-01b7-4c32-93be-dcb96b6b3b42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:44.948837 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:44.948791    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-data" pod "d4b7a180-01b7-4c32-93be-dcb96b6b3b42" (UID: "d4b7a180-01b7-4c32-93be-dcb96b6b3b42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:44.948969 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:44.948940    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:44.948969 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:44.948959    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:44.949060 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:44.948974    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:44.949060 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:44.948988    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:44.949201 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:44.949180    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-credential-socket" pod "d4b7a180-01b7-4c32-93be-dcb96b6b3b42" (UID: "d4b7a180-01b7-4c32-93be-dcb96b6b3b42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:44.950970 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:44.950943    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-token" pod "d4b7a180-01b7-4c32-93be-dcb96b6b3b42" (UID: "d4b7a180-01b7-4c32-93be-dcb96b6b3b42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:44.951146 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:44.951121    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-kube-api-access-xgcd4" pod "d4b7a180-01b7-4c32-93be-dcb96b6b3b42" (UID: "d4b7a180-01b7-4c32-93be-dcb96b6b3b42"). InnerVolumeSpecName "kube-api-access-xgcd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:44.951437 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:44.951419    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-envoy" pod "d4b7a180-01b7-4c32-93be-dcb96b6b3b42" (UID: "d4b7a180-01b7-4c32-93be-dcb96b6b3b42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:44.951544 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:44.951524    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-podinfo" pod "d4b7a180-01b7-4c32-93be-dcb96b6b3b42" (UID: "d4b7a180-01b7-4c32-93be-dcb96b6b3b42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:07:45.050192 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:45.050156    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgcd4\" (UniqueName: \"kubernetes.io/projected/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-kube-api-access-xgcd4\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:45.050192 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:45.050194    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:45.050192 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:45.050208    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:45.050569 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:45.050260    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:45.050569 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:45.050275    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b7a180-01b7-4c32-93be-dcb96b6b3b42-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:45.455525 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:45.455484    5875 generic.go:356] "Generic (PLEG): container finished" podID="d4b7a180-01b7-4c32-93be-dcb96b6b3b42" containerID="f0a92bdc72fd824df97fc3528a7b6276d955dc32e5d8f2db4f91f239bdb8ba46" exitCode=0
Nov 02 02:07:45.455709 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:45.455534    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"
Nov 02 02:07:45.455709 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:45.455565    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf" event={"ID":"d4b7a180-01b7-4c32-93be-dcb96b6b3b42","Type":"ContainerDied","Data":"f0a92bdc72fd824df97fc3528a7b6276d955dc32e5d8f2db4f91f239bdb8ba46"}
Nov 02 02:07:45.455709 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:45.455620    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf" event={"ID":"d4b7a180-01b7-4c32-93be-dcb96b6b3b42","Type":"ContainerDied","Data":"a60e052c328432e3bf3593d373f27b9f11a2da2359e3c9bb309d2f9cbb238879"}
Nov 02 02:07:45.455709 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:45.455641    5875 scope.go:122] "RemoveContainer" containerID="f0a92bdc72fd824df97fc3528a7b6276d955dc32e5d8f2db4f91f239bdb8ba46"
Nov 02 02:07:45.467518 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:45.467502    5875 scope.go:122] "RemoveContainer" containerID="639b38cd3662735e99dce69faf3e49b82f5b265b24507010e1ef42dc813016a9"
Nov 02 02:07:45.476816 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:45.476796    5875 scope.go:122] "RemoveContainer" containerID="f0a92bdc72fd824df97fc3528a7b6276d955dc32e5d8f2db4f91f239bdb8ba46"
Nov 02 02:07:45.477077 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:07:45.477054    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0a92bdc72fd824df97fc3528a7b6276d955dc32e5d8f2db4f91f239bdb8ba46\": container with ID starting with f0a92bdc72fd824df97fc3528a7b6276d955dc32e5d8f2db4f91f239bdb8ba46 not found: ID does not exist" containerID="f0a92bdc72fd824df97fc3528a7b6276d955dc32e5d8f2db4f91f239bdb8ba46"
Nov 02 02:07:45.477135 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:45.477086    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0a92bdc72fd824df97fc3528a7b6276d955dc32e5d8f2db4f91f239bdb8ba46"} err="failed to get container status \"f0a92bdc72fd824df97fc3528a7b6276d955dc32e5d8f2db4f91f239bdb8ba46\": rpc error: code = NotFound desc = could not find container \"f0a92bdc72fd824df97fc3528a7b6276d955dc32e5d8f2db4f91f239bdb8ba46\": container with ID starting with f0a92bdc72fd824df97fc3528a7b6276d955dc32e5d8f2db4f91f239bdb8ba46 not found: ID does not exist"
Nov 02 02:07:45.477135 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:45.477105    5875 scope.go:122] "RemoveContainer" containerID="639b38cd3662735e99dce69faf3e49b82f5b265b24507010e1ef42dc813016a9"
Nov 02 02:07:45.477397 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:07:45.477379    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"639b38cd3662735e99dce69faf3e49b82f5b265b24507010e1ef42dc813016a9\": container with ID starting with 639b38cd3662735e99dce69faf3e49b82f5b265b24507010e1ef42dc813016a9 not found: ID does not exist" containerID="639b38cd3662735e99dce69faf3e49b82f5b265b24507010e1ef42dc813016a9"
Nov 02 02:07:45.477457 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:45.477403    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639b38cd3662735e99dce69faf3e49b82f5b265b24507010e1ef42dc813016a9"} err="failed to get container status \"639b38cd3662735e99dce69faf3e49b82f5b265b24507010e1ef42dc813016a9\": rpc error: code = NotFound desc = could not find container \"639b38cd3662735e99dce69faf3e49b82f5b265b24507010e1ef42dc813016a9\": container with ID starting with 639b38cd3662735e99dce69faf3e49b82f5b265b24507010e1ef42dc813016a9 not found: ID does not exist"
Nov 02 02:07:45.620567 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:45.620534    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"]
Nov 02 02:07:45.670366 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:45.670337    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-73e1cc-openshift-default-7b887c446f-bntcf"]
Nov 02 02:07:46.693974 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.693908    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:07:46.697892 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.697867    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4b7a180-01b7-4c32-93be-dcb96b6b3b42" path="/var/lib/kubelet/pods/d4b7a180-01b7-4c32-93be-dcb96b6b3b42/volumes"
Nov 02 02:07:46.771611 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.771584    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c5779f-1d48-4749-bcb0-8313b202142d-kube-api-access-6kczh\" (UniqueName: \"kubernetes.io/projected/a0c5779f-1d48-4749-bcb0-8313b202142d-kube-api-access-6kczh\") pod \"a0c5779f-1d48-4749-bcb0-8313b202142d\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") "
Nov 02 02:07:46.771762 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.771679    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0c5779f-1d48-4749-bcb0-8313b202142d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0c5779f-1d48-4749-bcb0-8313b202142d-istiod-ca-cert\") pod \"a0c5779f-1d48-4749-bcb0-8313b202142d\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") "
Nov 02 02:07:46.771762 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.771720    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-envoy\") pod \"a0c5779f-1d48-4749-bcb0-8313b202142d\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") "
Nov 02 02:07:46.771762 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.771756    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-workload-certs\") pod \"a0c5779f-1d48-4749-bcb0-8313b202142d\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") "
Nov 02 02:07:46.771936 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.771860    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-workload-socket\") pod \"a0c5779f-1d48-4749-bcb0-8313b202142d\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") "
Nov 02 02:07:46.771936 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.771888    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-token\" (UniqueName: \"kubernetes.io/projected/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-token\") pod \"a0c5779f-1d48-4749-bcb0-8313b202142d\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") "
Nov 02 02:07:46.771936 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.771921    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-data\") pod \"a0c5779f-1d48-4749-bcb0-8313b202142d\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") "
Nov 02 02:07:46.772110 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.771969    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-credential-socket\") pod \"a0c5779f-1d48-4749-bcb0-8313b202142d\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") "
Nov 02 02:07:46.772110 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.772001    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-workload-certs" pod "a0c5779f-1d48-4749-bcb0-8313b202142d" (UID: "a0c5779f-1d48-4749-bcb0-8313b202142d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:46.772110 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.772011    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-podinfo\") pod \"a0c5779f-1d48-4749-bcb0-8313b202142d\" (UID: \"a0c5779f-1d48-4749-bcb0-8313b202142d\") "
Nov 02 02:07:46.772311 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.772119    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-workload-socket" pod "a0c5779f-1d48-4749-bcb0-8313b202142d" (UID: "a0c5779f-1d48-4749-bcb0-8313b202142d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:46.772311 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.772133    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c5779f-1d48-4749-bcb0-8313b202142d-istiod-ca-cert" pod "a0c5779f-1d48-4749-bcb0-8313b202142d" (UID: "a0c5779f-1d48-4749-bcb0-8313b202142d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:46.772311 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.772197    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-data" pod "a0c5779f-1d48-4749-bcb0-8313b202142d" (UID: "a0c5779f-1d48-4749-bcb0-8313b202142d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:46.772638 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.772382    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-credential-socket" pod "a0c5779f-1d48-4749-bcb0-8313b202142d" (UID: "a0c5779f-1d48-4749-bcb0-8313b202142d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:46.772810 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.772771    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0c5779f-1d48-4749-bcb0-8313b202142d-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:46.772810 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.772799    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:46.772945 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.772814    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:46.772945 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.772829    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:46.772945 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.772851    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:46.774165 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.774133    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c5779f-1d48-4749-bcb0-8313b202142d-kube-api-access-6kczh" pod "a0c5779f-1d48-4749-bcb0-8313b202142d" (UID: "a0c5779f-1d48-4749-bcb0-8313b202142d"). InnerVolumeSpecName "kube-api-access-6kczh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:46.774300 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.774205    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-token" pod "a0c5779f-1d48-4749-bcb0-8313b202142d" (UID: "a0c5779f-1d48-4749-bcb0-8313b202142d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:46.774495 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.774475    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-envoy" pod "a0c5779f-1d48-4749-bcb0-8313b202142d" (UID: "a0c5779f-1d48-4749-bcb0-8313b202142d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:46.774682 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.774665    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-podinfo" pod "a0c5779f-1d48-4749-bcb0-8313b202142d" (UID: "a0c5779f-1d48-4749-bcb0-8313b202142d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:07:46.873926 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.873887    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:46.873926 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.873918    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:46.873926 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.873931    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0c5779f-1d48-4749-bcb0-8313b202142d-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:46.874167 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:46.873944    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kczh\" (UniqueName: \"kubernetes.io/projected/a0c5779f-1d48-4749-bcb0-8313b202142d-kube-api-access-6kczh\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:47.410652 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:47.410614    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz"]
Nov 02 02:07:47.410853 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:47.410825    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz" podUID="99df19ae-03ba-4b87-ae0d-8d837d693d7d" containerName="maas-api" containerID="cri-o://1a7e3350eb4cef38949f56de73a434e350c8331003f99d5186f83042a4a072bb" gracePeriod=30
Nov 02 02:07:47.414841 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:47.414813    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7"]
Nov 02 02:07:47.509381 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:47.508012    5875 generic.go:356] "Generic (PLEG): container finished" podID="a0c5779f-1d48-4749-bcb0-8313b202142d" containerID="489d1950ec1c74f0377e702f56c7b8ab402593a229a78f42cc4b4c783bcdf856" exitCode=0
Nov 02 02:07:47.509381 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:47.508130    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t" event={"ID":"a0c5779f-1d48-4749-bcb0-8313b202142d","Type":"ContainerDied","Data":"489d1950ec1c74f0377e702f56c7b8ab402593a229a78f42cc4b4c783bcdf856"}
Nov 02 02:07:47.509381 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:47.508213    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t" event={"ID":"a0c5779f-1d48-4749-bcb0-8313b202142d","Type":"ContainerDied","Data":"a7ae64c9939b0731e45a454ef3650c5f6d3c802a9c2bfef6a26b7caee9ab0c4b"}
Nov 02 02:07:47.509381 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:47.508268    5875 scope.go:122] "RemoveContainer" containerID="489d1950ec1c74f0377e702f56c7b8ab402593a229a78f42cc4b4c783bcdf856"
Nov 02 02:07:47.509381 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:47.508470    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"
Nov 02 02:07:47.534186 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:47.534148    5875 scope.go:122] "RemoveContainer" containerID="547d14704548a99f3eaf569e2fe5da0ee2b16399bd0d050fd3ea68cee2438cdf"
Nov 02 02:07:47.544178 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:47.543961    5875 scope.go:122] "RemoveContainer" containerID="489d1950ec1c74f0377e702f56c7b8ab402593a229a78f42cc4b4c783bcdf856"
Nov 02 02:07:47.544400 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:07:47.544361    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"489d1950ec1c74f0377e702f56c7b8ab402593a229a78f42cc4b4c783bcdf856\": container with ID starting with 489d1950ec1c74f0377e702f56c7b8ab402593a229a78f42cc4b4c783bcdf856 not found: ID does not exist" containerID="489d1950ec1c74f0377e702f56c7b8ab402593a229a78f42cc4b4c783bcdf856"
Nov 02 02:07:47.544527 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:47.544409    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489d1950ec1c74f0377e702f56c7b8ab402593a229a78f42cc4b4c783bcdf856"} err="failed to get container status \"489d1950ec1c74f0377e702f56c7b8ab402593a229a78f42cc4b4c783bcdf856\": rpc error: code = NotFound desc = could not find container \"489d1950ec1c74f0377e702f56c7b8ab402593a229a78f42cc4b4c783bcdf856\": container with ID starting with 489d1950ec1c74f0377e702f56c7b8ab402593a229a78f42cc4b4c783bcdf856 not found: ID does not exist"
Nov 02 02:07:47.544527 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:47.544437    5875 scope.go:122] "RemoveContainer" containerID="547d14704548a99f3eaf569e2fe5da0ee2b16399bd0d050fd3ea68cee2438cdf"
Nov 02 02:07:47.544764 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:07:47.544736    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"547d14704548a99f3eaf569e2fe5da0ee2b16399bd0d050fd3ea68cee2438cdf\": container with ID starting with 547d14704548a99f3eaf569e2fe5da0ee2b16399bd0d050fd3ea68cee2438cdf not found: ID does not exist" containerID="547d14704548a99f3eaf569e2fe5da0ee2b16399bd0d050fd3ea68cee2438cdf"
Nov 02 02:07:47.544829 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:47.544782    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"547d14704548a99f3eaf569e2fe5da0ee2b16399bd0d050fd3ea68cee2438cdf"} err="failed to get container status \"547d14704548a99f3eaf569e2fe5da0ee2b16399bd0d050fd3ea68cee2438cdf\": rpc error: code = NotFound desc = could not find container \"547d14704548a99f3eaf569e2fe5da0ee2b16399bd0d050fd3ea68cee2438cdf\": container with ID starting with 547d14704548a99f3eaf569e2fe5da0ee2b16399bd0d050fd3ea68cee2438cdf not found: ID does not exist"
Nov 02 02:07:47.766730 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:47.766697    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"]
Nov 02 02:07:47.774269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:47.772618    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz"
Nov 02 02:07:47.777296 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:47.777273    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-129de4-openshift-default-577874cbbf-9xq8t"]
Nov 02 02:07:47.908483 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:47.908452    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99df19ae-03ba-4b87-ae0d-8d837d693d7d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99df19ae-03ba-4b87-ae0d-8d837d693d7d-openshift-service-ca\") pod \"99df19ae-03ba-4b87-ae0d-8d837d693d7d\" (UID: \"99df19ae-03ba-4b87-ae0d-8d837d693d7d\") "
Nov 02 02:07:47.908742 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:47.908723    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99df19ae-03ba-4b87-ae0d-8d837d693d7d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99df19ae-03ba-4b87-ae0d-8d837d693d7d-maas-api-tls\") pod \"99df19ae-03ba-4b87-ae0d-8d837d693d7d\" (UID: \"99df19ae-03ba-4b87-ae0d-8d837d693d7d\") "
Nov 02 02:07:47.908788 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:47.908766    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99df19ae-03ba-4b87-ae0d-8d837d693d7d-kube-api-access-6q6zv\" (UniqueName: \"kubernetes.io/projected/99df19ae-03ba-4b87-ae0d-8d837d693d7d-kube-api-access-6q6zv\") pod \"99df19ae-03ba-4b87-ae0d-8d837d693d7d\" (UID: \"99df19ae-03ba-4b87-ae0d-8d837d693d7d\") "
Nov 02 02:07:47.908788 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:47.908770    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99df19ae-03ba-4b87-ae0d-8d837d693d7d-openshift-service-ca" pod "99df19ae-03ba-4b87-ae0d-8d837d693d7d" (UID: "99df19ae-03ba-4b87-ae0d-8d837d693d7d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:47.908876 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:47.908806    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99df19ae-03ba-4b87-ae0d-8d837d693d7d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/99df19ae-03ba-4b87-ae0d-8d837d693d7d-metrics-certs\") pod \"99df19ae-03ba-4b87-ae0d-8d837d693d7d\" (UID: \"99df19ae-03ba-4b87-ae0d-8d837d693d7d\") "
Nov 02 02:07:47.909405 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:47.909370    5875 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99df19ae-03ba-4b87-ae0d-8d837d693d7d-openshift-service-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:47.911121 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:47.911094    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99df19ae-03ba-4b87-ae0d-8d837d693d7d-metrics-certs" pod "99df19ae-03ba-4b87-ae0d-8d837d693d7d" (UID: "99df19ae-03ba-4b87-ae0d-8d837d693d7d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:07:47.911634 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:47.911614    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99df19ae-03ba-4b87-ae0d-8d837d693d7d-kube-api-access-6q6zv" pod "99df19ae-03ba-4b87-ae0d-8d837d693d7d" (UID: "99df19ae-03ba-4b87-ae0d-8d837d693d7d"). InnerVolumeSpecName "kube-api-access-6q6zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:47.911716 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:47.911634    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99df19ae-03ba-4b87-ae0d-8d837d693d7d-maas-api-tls" pod "99df19ae-03ba-4b87-ae0d-8d837d693d7d" (UID: "99df19ae-03ba-4b87-ae0d-8d837d693d7d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:07:48.030753 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:48.030610    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99df19ae-03ba-4b87-ae0d-8d837d693d7d-maas-api-tls\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:48.030753 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:48.030655    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q6zv\" (UniqueName: \"kubernetes.io/projected/99df19ae-03ba-4b87-ae0d-8d837d693d7d-kube-api-access-6q6zv\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:48.030753 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:48.030675    5875 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99df19ae-03ba-4b87-ae0d-8d837d693d7d-metrics-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:48.538651 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:48.538417    5875 generic.go:356] "Generic (PLEG): container finished" podID="99df19ae-03ba-4b87-ae0d-8d837d693d7d" containerID="1a7e3350eb4cef38949f56de73a434e350c8331003f99d5186f83042a4a072bb" exitCode=0
Nov 02 02:07:48.538651 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:48.538597    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz" event={"ID":"99df19ae-03ba-4b87-ae0d-8d837d693d7d","Type":"ContainerDied","Data":"1a7e3350eb4cef38949f56de73a434e350c8331003f99d5186f83042a4a072bb"}
Nov 02 02:07:48.538651 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:48.538636    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz" event={"ID":"99df19ae-03ba-4b87-ae0d-8d837d693d7d","Type":"ContainerDied","Data":"3708071aa8f98b7b286618412e4ead368effc75e35b8c254ee94e2f8705d5ddf"}
Nov 02 02:07:48.538651 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:48.538656    5875 scope.go:122] "RemoveContainer" containerID="1a7e3350eb4cef38949f56de73a434e350c8331003f99d5186f83042a4a072bb"
Nov 02 02:07:48.539077 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:48.538732    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz"
Nov 02 02:07:48.573318 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:48.572200    5875 scope.go:122] "RemoveContainer" containerID="1a7e3350eb4cef38949f56de73a434e350c8331003f99d5186f83042a4a072bb"
Nov 02 02:07:48.573655 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:07:48.573572    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a7e3350eb4cef38949f56de73a434e350c8331003f99d5186f83042a4a072bb\": container with ID starting with 1a7e3350eb4cef38949f56de73a434e350c8331003f99d5186f83042a4a072bb not found: ID does not exist" containerID="1a7e3350eb4cef38949f56de73a434e350c8331003f99d5186f83042a4a072bb"
Nov 02 02:07:48.573655 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:48.573614    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a7e3350eb4cef38949f56de73a434e350c8331003f99d5186f83042a4a072bb"} err="failed to get container status \"1a7e3350eb4cef38949f56de73a434e350c8331003f99d5186f83042a4a072bb\": rpc error: code = NotFound desc = could not find container \"1a7e3350eb4cef38949f56de73a434e350c8331003f99d5186f83042a4a072bb\": container with ID starting with 1a7e3350eb4cef38949f56de73a434e350c8331003f99d5186f83042a4a072bb not found: ID does not exist"
Nov 02 02:07:48.695521 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:48.695494    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c5779f-1d48-4749-bcb0-8313b202142d" path="/var/lib/kubelet/pods/a0c5779f-1d48-4749-bcb0-8313b202142d/volumes"
Nov 02 02:07:48.713005 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:48.712980    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz"]
Nov 02 02:07:48.718411 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:48.718386    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c1aad5-54d5f4b59-57pcz"]
Nov 02 02:07:50.695992 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:50.695961    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99df19ae-03ba-4b87-ae0d-8d837d693d7d" path="/var/lib/kubelet/pods/99df19ae-03ba-4b87-ae0d-8d837d693d7d/volumes"
Nov 02 02:07:51.986153 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:51.986117    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj"]
Nov 02 02:07:52.091647 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:52.091616    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7"]
Nov 02 02:07:52.428564 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:52.428435    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7" podUID="0c003811-1913-436c-b5c2-5e67bc7b72a1" containerName="payload-processing" containerID="cri-o://4e7d61ecb78d274040844a90e86dc13a0279f791eefcf0d099e0bde3b58bff12" gracePeriod=25
Nov 02 02:07:52.586378 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:52.586347    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:07:54.485190 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:54.485156    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz"]
Nov 02 02:07:54.590761 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:54.590722    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k"]
Nov 02 02:07:55.401296 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:55.401258    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" podUID="6e39fc3d-d730-4df7-8d10-2376656a88d2" containerName="main" containerID="cri-o://9e8b080b794a240e71ef7100917d80541d46fe09759aa38a53cd1318ea031730" gracePeriod=45
Nov 02 02:07:55.618159 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:55.618124    5875 generic.go:356] "Generic (PLEG): container finished" podID="6e39fc3d-d730-4df7-8d10-2376656a88d2" containerID="9e8b080b794a240e71ef7100917d80541d46fe09759aa38a53cd1318ea031730" exitCode=0
Nov 02 02:07:55.618655 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:55.618176    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" event={"ID":"6e39fc3d-d730-4df7-8d10-2376656a88d2","Type":"ContainerDied","Data":"9e8b080b794a240e71ef7100917d80541d46fe09759aa38a53cd1318ea031730"}
Nov 02 02:07:55.675161 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:55.675135    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 02:07:55.859771 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:55.859736    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-model-cache\") pod \"6e39fc3d-d730-4df7-8d10-2376656a88d2\" (UID: \"6e39fc3d-d730-4df7-8d10-2376656a88d2\") "
Nov 02 02:07:55.859932 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:55.859813    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e39fc3d-d730-4df7-8d10-2376656a88d2-kube-api-access-b8rkk\" (UniqueName: \"kubernetes.io/projected/6e39fc3d-d730-4df7-8d10-2376656a88d2-kube-api-access-b8rkk\") pod \"6e39fc3d-d730-4df7-8d10-2376656a88d2\" (UID: \"6e39fc3d-d730-4df7-8d10-2376656a88d2\") "
Nov 02 02:07:55.859932 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:55.859851    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-dshm\") pod \"6e39fc3d-d730-4df7-8d10-2376656a88d2\" (UID: \"6e39fc3d-d730-4df7-8d10-2376656a88d2\") "
Nov 02 02:07:55.860021 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:55.859944    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-home\" (UniqueName: \"kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-home\") pod \"6e39fc3d-d730-4df7-8d10-2376656a88d2\" (UID: \"6e39fc3d-d730-4df7-8d10-2376656a88d2\") "
Nov 02 02:07:55.860021 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:55.859991    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-model-cache" pod "6e39fc3d-d730-4df7-8d10-2376656a88d2" (UID: "6e39fc3d-d730-4df7-8d10-2376656a88d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:55.860021 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:55.860008    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e39fc3d-d730-4df7-8d10-2376656a88d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e39fc3d-d730-4df7-8d10-2376656a88d2-tls-certs\") pod \"6e39fc3d-d730-4df7-8d10-2376656a88d2\" (UID: \"6e39fc3d-d730-4df7-8d10-2376656a88d2\") "
Nov 02 02:07:55.860180 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:55.860076    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-tmp-dir\") pod \"6e39fc3d-d730-4df7-8d10-2376656a88d2\" (UID: \"6e39fc3d-d730-4df7-8d10-2376656a88d2\") "
Nov 02 02:07:55.860317 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:55.860294    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-home" pod "6e39fc3d-d730-4df7-8d10-2376656a88d2" (UID: "6e39fc3d-d730-4df7-8d10-2376656a88d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:55.860409 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:55.860313    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-tmp-dir" pod "6e39fc3d-d730-4df7-8d10-2376656a88d2" (UID: "6e39fc3d-d730-4df7-8d10-2376656a88d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:55.860751 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:55.860662    5875 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-model-cache\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:55.860751 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:55.860685    5875 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-home\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:55.860751 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:55.860700    5875 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-tmp-dir\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:55.862329 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:55.862305    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e39fc3d-d730-4df7-8d10-2376656a88d2-tls-certs" pod "6e39fc3d-d730-4df7-8d10-2376656a88d2" (UID: "6e39fc3d-d730-4df7-8d10-2376656a88d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:07:55.862437 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:55.862379    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e39fc3d-d730-4df7-8d10-2376656a88d2-kube-api-access-b8rkk" pod "6e39fc3d-d730-4df7-8d10-2376656a88d2" (UID: "6e39fc3d-d730-4df7-8d10-2376656a88d2"). InnerVolumeSpecName "kube-api-access-b8rkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:55.862437 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:55.862386    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-dshm" pod "6e39fc3d-d730-4df7-8d10-2376656a88d2" (UID: "6e39fc3d-d730-4df7-8d10-2376656a88d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:55.886608 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:55.886579    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7" podUID="0c003811-1913-436c-b5c2-5e67bc7b72a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:07:55.962047 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:55.962018    5875 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e39fc3d-d730-4df7-8d10-2376656a88d2-tls-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:55.962047 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:55.962046    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8rkk\" (UniqueName: \"kubernetes.io/projected/6e39fc3d-d730-4df7-8d10-2376656a88d2-kube-api-access-b8rkk\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:55.962304 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:55.962056    5875 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e39fc3d-d730-4df7-8d10-2376656a88d2-dshm\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:56.626583 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:56.626557    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"
Nov 02 02:07:56.627061 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:56.626605    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx" event={"ID":"6e39fc3d-d730-4df7-8d10-2376656a88d2","Type":"ContainerDied","Data":"e27ca9db4263a819edf97972f350144c43ffedfcecf1e4aa72dd48afde24306e"}
Nov 02 02:07:56.627061 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:56.626664    5875 scope.go:122] "RemoveContainer" containerID="9e8b080b794a240e71ef7100917d80541d46fe09759aa38a53cd1318ea031730"
Nov 02 02:07:56.791187 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:56.791157    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"]
Nov 02 02:07:56.840091 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:56.840064    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ed0600/premium-w4-ed0600-kserve-788774b5dc-lzntx"]
Nov 02 02:07:56.946129 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:56.946052    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"]
Nov 02 02:07:56.946342 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:56.946303    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg" podUID="d1dafba9-1caf-4355-bfa1-a3b8cea9155e" containerName="istio-proxy" containerID="cri-o://969612c799fd017191d24f85bce96313458da87cffb21d5fda1afd3d27105b2c" gracePeriod=30
Nov 02 02:07:56.986922 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:56.986859    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj" podUID="32eed2d4-07e6-41ba-885c-438ddfe308c7" containerName="payload-processing" containerID="cri-o://1d06727cb9ffb0526fafdbefcb0b1104bfd4904e4fe69806f2aee7002c1a960a" gracePeriod=25
Nov 02 02:07:57.108377 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.108308    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7" podUID="ade83e3d-3ae8-4732-9a85-955c54135ead" containerName="payload-pre-processing" containerID="cri-o://59557de22a314f21b060ba06168ee32eb40d862fcbbea6c08c0ed211cdd81821" gracePeriod=25
Nov 02 02:07:57.318177 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.318147    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj"
Nov 02 02:07:57.392428 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.392404    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32eed2d4-07e6-41ba-885c-438ddfe308c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32eed2d4-07e6-41ba-885c-438ddfe308c7-plugins-config-volume\") pod \"32eed2d4-07e6-41ba-885c-438ddfe308c7\" (UID: \"32eed2d4-07e6-41ba-885c-438ddfe308c7\") "
Nov 02 02:07:57.392565 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.392477    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32eed2d4-07e6-41ba-885c-438ddfe308c7-kube-api-access-msz7b\" (UniqueName: \"kubernetes.io/projected/32eed2d4-07e6-41ba-885c-438ddfe308c7-kube-api-access-msz7b\") pod \"32eed2d4-07e6-41ba-885c-438ddfe308c7\" (UID: \"32eed2d4-07e6-41ba-885c-438ddfe308c7\") "
Nov 02 02:07:57.392739 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.392716    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32eed2d4-07e6-41ba-885c-438ddfe308c7-plugins-config-volume" pod "32eed2d4-07e6-41ba-885c-438ddfe308c7" (UID: "32eed2d4-07e6-41ba-885c-438ddfe308c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:57.392896 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.392883    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32eed2d4-07e6-41ba-885c-438ddfe308c7-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:57.393386 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.393365    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7"
Nov 02 02:07:57.394621 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.394602    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32eed2d4-07e6-41ba-885c-438ddfe308c7-kube-api-access-msz7b" pod "32eed2d4-07e6-41ba-885c-438ddfe308c7" (UID: "32eed2d4-07e6-41ba-885c-438ddfe308c7"). InnerVolumeSpecName "kube-api-access-msz7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:57.493623 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.493579    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ade83e3d-3ae8-4732-9a85-955c54135ead-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade83e3d-3ae8-4732-9a85-955c54135ead-plugins-config-volume\") pod \"ade83e3d-3ae8-4732-9a85-955c54135ead\" (UID: \"ade83e3d-3ae8-4732-9a85-955c54135ead\") "
Nov 02 02:07:57.493840 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.493821    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade83e3d-3ae8-4732-9a85-955c54135ead-kube-api-access-qjbfv\" (UniqueName: \"kubernetes.io/projected/ade83e3d-3ae8-4732-9a85-955c54135ead-kube-api-access-qjbfv\") pod \"ade83e3d-3ae8-4732-9a85-955c54135ead\" (UID: \"ade83e3d-3ae8-4732-9a85-955c54135ead\") "
Nov 02 02:07:57.493998 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.493971    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade83e3d-3ae8-4732-9a85-955c54135ead-plugins-config-volume" pod "ade83e3d-3ae8-4732-9a85-955c54135ead" (UID: "ade83e3d-3ae8-4732-9a85-955c54135ead"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:57.494371 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.494350    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msz7b\" (UniqueName: \"kubernetes.io/projected/32eed2d4-07e6-41ba-885c-438ddfe308c7-kube-api-access-msz7b\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:57.494371 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.494369    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade83e3d-3ae8-4732-9a85-955c54135ead-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:57.496204 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.496183    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade83e3d-3ae8-4732-9a85-955c54135ead-kube-api-access-qjbfv" pod "ade83e3d-3ae8-4732-9a85-955c54135ead" (UID: "ade83e3d-3ae8-4732-9a85-955c54135ead"). InnerVolumeSpecName "kube-api-access-qjbfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:57.595564 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.595532    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjbfv\" (UniqueName: \"kubernetes.io/projected/ade83e3d-3ae8-4732-9a85-955c54135ead-kube-api-access-qjbfv\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:57.635643 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.635606    5875 generic.go:356] "Generic (PLEG): container finished" podID="32eed2d4-07e6-41ba-885c-438ddfe308c7" containerID="1d06727cb9ffb0526fafdbefcb0b1104bfd4904e4fe69806f2aee7002c1a960a" exitCode=0
Nov 02 02:07:57.636044 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.635655    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj"
Nov 02 02:07:57.636044 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.635698    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj" event={"ID":"32eed2d4-07e6-41ba-885c-438ddfe308c7","Type":"ContainerDied","Data":"1d06727cb9ffb0526fafdbefcb0b1104bfd4904e4fe69806f2aee7002c1a960a"}
Nov 02 02:07:57.636044 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.635771    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj" event={"ID":"32eed2d4-07e6-41ba-885c-438ddfe308c7","Type":"ContainerDied","Data":"df22a917258e9f29236a9128ab08342eb94fe18f3eca8696a225233ddeeed259"}
Nov 02 02:07:57.636044 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.635791    5875 scope.go:122] "RemoveContainer" containerID="1d06727cb9ffb0526fafdbefcb0b1104bfd4904e4fe69806f2aee7002c1a960a"
Nov 02 02:07:57.637118 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.637093    5875 generic.go:356] "Generic (PLEG): container finished" podID="ade83e3d-3ae8-4732-9a85-955c54135ead" containerID="59557de22a314f21b060ba06168ee32eb40d862fcbbea6c08c0ed211cdd81821" exitCode=0
Nov 02 02:07:57.637433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.637125    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7" event={"ID":"ade83e3d-3ae8-4732-9a85-955c54135ead","Type":"ContainerDied","Data":"59557de22a314f21b060ba06168ee32eb40d862fcbbea6c08c0ed211cdd81821"}
Nov 02 02:07:57.637433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.637140    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7"
Nov 02 02:07:57.637433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.637168    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7" event={"ID":"ade83e3d-3ae8-4732-9a85-955c54135ead","Type":"ContainerDied","Data":"88b11d86c5a2c605fed8fa3bcc7874f071a646219469127df7d11ff861ed7273"}
Nov 02 02:07:57.646534 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.646515    5875 scope.go:122] "RemoveContainer" containerID="1d06727cb9ffb0526fafdbefcb0b1104bfd4904e4fe69806f2aee7002c1a960a"
Nov 02 02:07:57.646948 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:07:57.646814    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d06727cb9ffb0526fafdbefcb0b1104bfd4904e4fe69806f2aee7002c1a960a\": container with ID starting with 1d06727cb9ffb0526fafdbefcb0b1104bfd4904e4fe69806f2aee7002c1a960a not found: ID does not exist" containerID="1d06727cb9ffb0526fafdbefcb0b1104bfd4904e4fe69806f2aee7002c1a960a"
Nov 02 02:07:57.646948 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.646846    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d06727cb9ffb0526fafdbefcb0b1104bfd4904e4fe69806f2aee7002c1a960a"} err="failed to get container status \"1d06727cb9ffb0526fafdbefcb0b1104bfd4904e4fe69806f2aee7002c1a960a\": rpc error: code = NotFound desc = could not find container \"1d06727cb9ffb0526fafdbefcb0b1104bfd4904e4fe69806f2aee7002c1a960a\": container with ID starting with 1d06727cb9ffb0526fafdbefcb0b1104bfd4904e4fe69806f2aee7002c1a960a not found: ID does not exist"
Nov 02 02:07:57.646948 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.646878    5875 scope.go:122] "RemoveContainer" containerID="59557de22a314f21b060ba06168ee32eb40d862fcbbea6c08c0ed211cdd81821"
Nov 02 02:07:57.656026 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.656004    5875 scope.go:122] "RemoveContainer" containerID="59557de22a314f21b060ba06168ee32eb40d862fcbbea6c08c0ed211cdd81821"
Nov 02 02:07:57.656292 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:07:57.656269    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59557de22a314f21b060ba06168ee32eb40d862fcbbea6c08c0ed211cdd81821\": container with ID starting with 59557de22a314f21b060ba06168ee32eb40d862fcbbea6c08c0ed211cdd81821 not found: ID does not exist" containerID="59557de22a314f21b060ba06168ee32eb40d862fcbbea6c08c0ed211cdd81821"
Nov 02 02:07:57.656368 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.656296    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59557de22a314f21b060ba06168ee32eb40d862fcbbea6c08c0ed211cdd81821"} err="failed to get container status \"59557de22a314f21b060ba06168ee32eb40d862fcbbea6c08c0ed211cdd81821\": rpc error: code = NotFound desc = could not find container \"59557de22a314f21b060ba06168ee32eb40d862fcbbea6c08c0ed211cdd81821\": container with ID starting with 59557de22a314f21b060ba06168ee32eb40d862fcbbea6c08c0ed211cdd81821 not found: ID does not exist"
Nov 02 02:07:57.799057 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.798979    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7"]
Nov 02 02:07:57.802495 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.802470    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8e79d7-6c6d5d8d8c-s68c7"]
Nov 02 02:07:57.912081 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.912046    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj"]
Nov 02 02:07:57.917887 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:57.917858    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-8e79d7-665699d565-l46gj"]
Nov 02 02:07:58.695769 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:58.695733    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32eed2d4-07e6-41ba-885c-438ddfe308c7" path="/var/lib/kubelet/pods/32eed2d4-07e6-41ba-885c-438ddfe308c7/volumes"
Nov 02 02:07:58.696379 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:58.696359    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e39fc3d-d730-4df7-8d10-2376656a88d2" path="/var/lib/kubelet/pods/6e39fc3d-d730-4df7-8d10-2376656a88d2/volumes"
Nov 02 02:07:58.696797 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:58.696777    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ade83e3d-3ae8-4732-9a85-955c54135ead" path="/var/lib/kubelet/pods/ade83e3d-3ae8-4732-9a85-955c54135ead/volumes"
Nov 02 02:07:59.486340 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:59.486267    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz" podUID="ce33230d-5eac-461b-b766-a40de8dda983" containerName="payload-processing" containerID="cri-o://c2437684137f1faf93b550e8fcb9bf7a245c72cb8924c3b58dd9bab1a87a457f" gracePeriod=25
Nov 02 02:07:59.604182 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:59.602040    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k" podUID="90fc1e25-7db0-4438-b278-07a551ac3314" containerName="payload-pre-processing" containerID="cri-o://3199de6764c8aa2b399dd447fd1e7c8c6f0c5b08f33b69d8224b01e6969178a7" gracePeriod=25
Nov 02 02:07:59.610501 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:07:59.610464    5875 logging.go:55] [core] [Channel #4677 SubChannel #4678]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.159:50052", ServerName: "10.134.0.159:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.159:50052: connect: connection refused"
Nov 02 02:07:59.653774 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:59.653727    5875 generic.go:356] "Generic (PLEG): container finished" podID="ce33230d-5eac-461b-b766-a40de8dda983" containerID="c2437684137f1faf93b550e8fcb9bf7a245c72cb8924c3b58dd9bab1a87a457f" exitCode=0
Nov 02 02:07:59.653925 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:59.653797    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz" event={"ID":"ce33230d-5eac-461b-b766-a40de8dda983","Type":"ContainerDied","Data":"c2437684137f1faf93b550e8fcb9bf7a245c72cb8924c3b58dd9bab1a87a457f"}
Nov 02 02:07:59.664915 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:59.664876    5875 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:07:59.665046 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:59.664947    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg" podUID="d1dafba9-1caf-4355-bfa1-a3b8cea9155e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:07:59.810169 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:59.810138    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz"
Nov 02 02:07:59.884074 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:59.884045    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k"
Nov 02 02:07:59.931456 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:59.931429    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90fc1e25-7db0-4438-b278-07a551ac3314-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90fc1e25-7db0-4438-b278-07a551ac3314-plugins-config-volume\") pod \"90fc1e25-7db0-4438-b278-07a551ac3314\" (UID: \"90fc1e25-7db0-4438-b278-07a551ac3314\") "
Nov 02 02:07:59.931620 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:59.931477    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce33230d-5eac-461b-b766-a40de8dda983-kube-api-access-xkvx8\" (UniqueName: \"kubernetes.io/projected/ce33230d-5eac-461b-b766-a40de8dda983-kube-api-access-xkvx8\") pod \"ce33230d-5eac-461b-b766-a40de8dda983\" (UID: \"ce33230d-5eac-461b-b766-a40de8dda983\") "
Nov 02 02:07:59.931692 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:59.931619    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce33230d-5eac-461b-b766-a40de8dda983-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce33230d-5eac-461b-b766-a40de8dda983-plugins-config-volume\") pod \"ce33230d-5eac-461b-b766-a40de8dda983\" (UID: \"ce33230d-5eac-461b-b766-a40de8dda983\") "
Nov 02 02:07:59.931762 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:59.931743    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90fc1e25-7db0-4438-b278-07a551ac3314-kube-api-access-khhc8\" (UniqueName: \"kubernetes.io/projected/90fc1e25-7db0-4438-b278-07a551ac3314-kube-api-access-khhc8\") pod \"90fc1e25-7db0-4438-b278-07a551ac3314\" (UID: \"90fc1e25-7db0-4438-b278-07a551ac3314\") "
Nov 02 02:07:59.931936 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:59.931804    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90fc1e25-7db0-4438-b278-07a551ac3314-plugins-config-volume" pod "90fc1e25-7db0-4438-b278-07a551ac3314" (UID: "90fc1e25-7db0-4438-b278-07a551ac3314"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:59.932029 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:59.932007    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce33230d-5eac-461b-b766-a40de8dda983-plugins-config-volume" pod "ce33230d-5eac-461b-b766-a40de8dda983" (UID: "ce33230d-5eac-461b-b766-a40de8dda983"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:07:59.932315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:59.932292    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90fc1e25-7db0-4438-b278-07a551ac3314-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:59.932401 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:59.932319    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce33230d-5eac-461b-b766-a40de8dda983-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:59.934507 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:59.934486    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90fc1e25-7db0-4438-b278-07a551ac3314-kube-api-access-khhc8" pod "90fc1e25-7db0-4438-b278-07a551ac3314" (UID: "90fc1e25-7db0-4438-b278-07a551ac3314"). InnerVolumeSpecName "kube-api-access-khhc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:59.934592 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:07:59.934507    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce33230d-5eac-461b-b766-a40de8dda983-kube-api-access-xkvx8" pod "ce33230d-5eac-461b-b766-a40de8dda983" (UID: "ce33230d-5eac-461b-b766-a40de8dda983"). InnerVolumeSpecName "kube-api-access-xkvx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:00.066143 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.065462    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkvx8\" (UniqueName: \"kubernetes.io/projected/ce33230d-5eac-461b-b766-a40de8dda983-kube-api-access-xkvx8\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:00.066143 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.065495    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khhc8\" (UniqueName: \"kubernetes.io/projected/90fc1e25-7db0-4438-b278-07a551ac3314-kube-api-access-khhc8\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:00.192156 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.192121    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6"]
Nov 02 02:08:00.192369 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.192339    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6" podUID="731b8642-8f48-42d2-82b1-10bece738f1b" containerName="maas-api" containerID="cri-o://9b7e70fc9fca734c913730e76c5a99fce5cbda3078e00530ddcf2b8d3cf54114" gracePeriod=30
Nov 02 02:08:00.467915 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.467887    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6"
Nov 02 02:08:00.571565 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.571534    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/731b8642-8f48-42d2-82b1-10bece738f1b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/731b8642-8f48-42d2-82b1-10bece738f1b-openshift-service-ca\") pod \"731b8642-8f48-42d2-82b1-10bece738f1b\" (UID: \"731b8642-8f48-42d2-82b1-10bece738f1b\") "
Nov 02 02:08:00.571721 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.571675    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/731b8642-8f48-42d2-82b1-10bece738f1b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/731b8642-8f48-42d2-82b1-10bece738f1b-maas-api-tls\") pod \"731b8642-8f48-42d2-82b1-10bece738f1b\" (UID: \"731b8642-8f48-42d2-82b1-10bece738f1b\") "
Nov 02 02:08:00.571793 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.571742    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/731b8642-8f48-42d2-82b1-10bece738f1b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/731b8642-8f48-42d2-82b1-10bece738f1b-metrics-certs\") pod \"731b8642-8f48-42d2-82b1-10bece738f1b\" (UID: \"731b8642-8f48-42d2-82b1-10bece738f1b\") "
Nov 02 02:08:00.571906 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.571880    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/731b8642-8f48-42d2-82b1-10bece738f1b-kube-api-access-9jvlw\" (UniqueName: \"kubernetes.io/projected/731b8642-8f48-42d2-82b1-10bece738f1b-kube-api-access-9jvlw\") pod \"731b8642-8f48-42d2-82b1-10bece738f1b\" (UID: \"731b8642-8f48-42d2-82b1-10bece738f1b\") "
Nov 02 02:08:00.571972 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.571941    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/731b8642-8f48-42d2-82b1-10bece738f1b-openshift-service-ca" pod "731b8642-8f48-42d2-82b1-10bece738f1b" (UID: "731b8642-8f48-42d2-82b1-10bece738f1b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:00.572464 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.572437    5875 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/731b8642-8f48-42d2-82b1-10bece738f1b-openshift-service-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:00.574443 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.574405    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/731b8642-8f48-42d2-82b1-10bece738f1b-maas-api-tls" pod "731b8642-8f48-42d2-82b1-10bece738f1b" (UID: "731b8642-8f48-42d2-82b1-10bece738f1b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:08:00.574829 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.574750    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731b8642-8f48-42d2-82b1-10bece738f1b-kube-api-access-9jvlw" pod "731b8642-8f48-42d2-82b1-10bece738f1b" (UID: "731b8642-8f48-42d2-82b1-10bece738f1b"). InnerVolumeSpecName "kube-api-access-9jvlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:00.574944 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.574845    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/731b8642-8f48-42d2-82b1-10bece738f1b-metrics-certs" pod "731b8642-8f48-42d2-82b1-10bece738f1b" (UID: "731b8642-8f48-42d2-82b1-10bece738f1b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:08:00.654781 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.624412    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz" podUID="ce33230d-5eac-461b-b766-a40de8dda983" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.159:50052\" within 1s: context deadline exceeded"
Nov 02 02:08:00.672449 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.672413    5875 generic.go:356] "Generic (PLEG): container finished" podID="731b8642-8f48-42d2-82b1-10bece738f1b" containerID="9b7e70fc9fca734c913730e76c5a99fce5cbda3078e00530ddcf2b8d3cf54114" exitCode=0
Nov 02 02:08:00.672640 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.672470    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6"
Nov 02 02:08:00.672640 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.672507    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6" event={"ID":"731b8642-8f48-42d2-82b1-10bece738f1b","Type":"ContainerDied","Data":"9b7e70fc9fca734c913730e76c5a99fce5cbda3078e00530ddcf2b8d3cf54114"}
Nov 02 02:08:00.672640 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.672572    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6" event={"ID":"731b8642-8f48-42d2-82b1-10bece738f1b","Type":"ContainerDied","Data":"c6826bda967e9354cb3871fb646ed8d821d06f13e4227c4f171bd73886156119"}
Nov 02 02:08:00.672640 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.672606    5875 scope.go:122] "RemoveContainer" containerID="9b7e70fc9fca734c913730e76c5a99fce5cbda3078e00530ddcf2b8d3cf54114"
Nov 02 02:08:00.673555 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.673528    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/731b8642-8f48-42d2-82b1-10bece738f1b-maas-api-tls\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:00.673696 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.673557    5875 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/731b8642-8f48-42d2-82b1-10bece738f1b-metrics-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:00.673696 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.673574    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jvlw\" (UniqueName: \"kubernetes.io/projected/731b8642-8f48-42d2-82b1-10bece738f1b-kube-api-access-9jvlw\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:00.675075 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.675048    5875 generic.go:356] "Generic (PLEG): container finished" podID="90fc1e25-7db0-4438-b278-07a551ac3314" containerID="3199de6764c8aa2b399dd447fd1e7c8c6f0c5b08f33b69d8224b01e6969178a7" exitCode=0
Nov 02 02:08:00.675265 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.675195    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k" event={"ID":"90fc1e25-7db0-4438-b278-07a551ac3314","Type":"ContainerDied","Data":"3199de6764c8aa2b399dd447fd1e7c8c6f0c5b08f33b69d8224b01e6969178a7"}
Nov 02 02:08:00.675355 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.675277    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k" event={"ID":"90fc1e25-7db0-4438-b278-07a551ac3314","Type":"ContainerDied","Data":"80fe2d2ea2ef3ba2478310946fde1d5d74e497ebc7791b99715cd20bca56bcae"}
Nov 02 02:08:00.675355 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.675344    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k"
Nov 02 02:08:00.684197 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.684174    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz" event={"ID":"ce33230d-5eac-461b-b766-a40de8dda983","Type":"ContainerDied","Data":"e45457eaaea897bfa73a3850f06376191434d10a64bffef004bcfea269b65324"}
Nov 02 02:08:00.684355 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.684297    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz"
Nov 02 02:08:00.692905 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.692876    5875 scope.go:122] "RemoveContainer" containerID="9b7e70fc9fca734c913730e76c5a99fce5cbda3078e00530ddcf2b8d3cf54114"
Nov 02 02:08:00.693285 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:08:00.693210    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b7e70fc9fca734c913730e76c5a99fce5cbda3078e00530ddcf2b8d3cf54114\": container with ID starting with 9b7e70fc9fca734c913730e76c5a99fce5cbda3078e00530ddcf2b8d3cf54114 not found: ID does not exist" containerID="9b7e70fc9fca734c913730e76c5a99fce5cbda3078e00530ddcf2b8d3cf54114"
Nov 02 02:08:00.693430 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.693294    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7e70fc9fca734c913730e76c5a99fce5cbda3078e00530ddcf2b8d3cf54114"} err="failed to get container status \"9b7e70fc9fca734c913730e76c5a99fce5cbda3078e00530ddcf2b8d3cf54114\": rpc error: code = NotFound desc = could not find container \"9b7e70fc9fca734c913730e76c5a99fce5cbda3078e00530ddcf2b8d3cf54114\": container with ID starting with 9b7e70fc9fca734c913730e76c5a99fce5cbda3078e00530ddcf2b8d3cf54114 not found: ID does not exist"
Nov 02 02:08:00.693430 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.693317    5875 scope.go:122] "RemoveContainer" containerID="3199de6764c8aa2b399dd447fd1e7c8c6f0c5b08f33b69d8224b01e6969178a7"
Nov 02 02:08:00.705664 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.705643    5875 scope.go:122] "RemoveContainer" containerID="3199de6764c8aa2b399dd447fd1e7c8c6f0c5b08f33b69d8224b01e6969178a7"
Nov 02 02:08:00.705925 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:08:00.705906    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3199de6764c8aa2b399dd447fd1e7c8c6f0c5b08f33b69d8224b01e6969178a7\": container with ID starting with 3199de6764c8aa2b399dd447fd1e7c8c6f0c5b08f33b69d8224b01e6969178a7 not found: ID does not exist" containerID="3199de6764c8aa2b399dd447fd1e7c8c6f0c5b08f33b69d8224b01e6969178a7"
Nov 02 02:08:00.705998 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.705936    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3199de6764c8aa2b399dd447fd1e7c8c6f0c5b08f33b69d8224b01e6969178a7"} err="failed to get container status \"3199de6764c8aa2b399dd447fd1e7c8c6f0c5b08f33b69d8224b01e6969178a7\": rpc error: code = NotFound desc = could not find container \"3199de6764c8aa2b399dd447fd1e7c8c6f0c5b08f33b69d8224b01e6969178a7\": container with ID starting with 3199de6764c8aa2b399dd447fd1e7c8c6f0c5b08f33b69d8224b01e6969178a7 not found: ID does not exist"
Nov 02 02:08:00.705998 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.705959    5875 scope.go:122] "RemoveContainer" containerID="c2437684137f1faf93b550e8fcb9bf7a245c72cb8924c3b58dd9bab1a87a457f"
Nov 02 02:08:00.875704 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.875671    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz"]
Nov 02 02:08:00.880388 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.880365    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-47eb54-5b569f674f-5bzkz"]
Nov 02 02:08:00.988280 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.988252    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k"]
Nov 02 02:08:00.992861 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:00.992833    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-47eb54-866bb94c9f-h289k"]
Nov 02 02:08:01.100873 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:01.100843    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6"]
Nov 02 02:08:01.106372 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:01.106260    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-jqrx6"]
Nov 02 02:08:02.229282 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.229209    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:08:02.302530 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.302492    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-podinfo\") pod \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") "
Nov 02 02:08:02.302776 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.302553    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istiod-ca-cert\") pod \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") "
Nov 02 02:08:02.302776 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.302639    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-workload-socket\") pod \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") "
Nov 02 02:08:02.302776 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.302700    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-data\") pod \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") "
Nov 02 02:08:02.302776 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.302763    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-credential-socket\") pod \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") "
Nov 02 02:08:02.305613 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.302795    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-envoy\") pod \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") "
Nov 02 02:08:02.305613 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.302842    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-token\" (UniqueName: \"kubernetes.io/projected/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-token\") pod \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") "
Nov 02 02:08:02.305613 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.302956    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-kube-api-access-52w5b\" (UniqueName: \"kubernetes.io/projected/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-kube-api-access-52w5b\") pod \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") "
Nov 02 02:08:02.305613 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.303033    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-workload-certs\") pod \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\" (UID: \"d1dafba9-1caf-4355-bfa1-a3b8cea9155e\") "
Nov 02 02:08:02.312190 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.312127    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istiod-ca-cert" pod "d1dafba9-1caf-4355-bfa1-a3b8cea9155e" (UID: "d1dafba9-1caf-4355-bfa1-a3b8cea9155e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:02.312438 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.312416    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-workload-socket" pod "d1dafba9-1caf-4355-bfa1-a3b8cea9155e" (UID: "d1dafba9-1caf-4355-bfa1-a3b8cea9155e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:02.312610 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.312590    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-credential-socket" pod "d1dafba9-1caf-4355-bfa1-a3b8cea9155e" (UID: "d1dafba9-1caf-4355-bfa1-a3b8cea9155e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:02.312846 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.312826    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-workload-certs" pod "d1dafba9-1caf-4355-bfa1-a3b8cea9155e" (UID: "d1dafba9-1caf-4355-bfa1-a3b8cea9155e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:02.313908 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.313867    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-data" pod "d1dafba9-1caf-4355-bfa1-a3b8cea9155e" (UID: "d1dafba9-1caf-4355-bfa1-a3b8cea9155e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:02.335755 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.319968    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:02.335755 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.320003    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:02.335755 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.320017    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:02.335755 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.320032    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:02.335755 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.320047    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:02.344268 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.344186    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-token" pod "d1dafba9-1caf-4355-bfa1-a3b8cea9155e" (UID: "d1dafba9-1caf-4355-bfa1-a3b8cea9155e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:02.344402 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.344385    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-envoy" pod "d1dafba9-1caf-4355-bfa1-a3b8cea9155e" (UID: "d1dafba9-1caf-4355-bfa1-a3b8cea9155e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:02.344474 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.344411    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-podinfo" pod "d1dafba9-1caf-4355-bfa1-a3b8cea9155e" (UID: "d1dafba9-1caf-4355-bfa1-a3b8cea9155e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:08:02.345734 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.345694    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-kube-api-access-52w5b" pod "d1dafba9-1caf-4355-bfa1-a3b8cea9155e" (UID: "d1dafba9-1caf-4355-bfa1-a3b8cea9155e"). InnerVolumeSpecName "kube-api-access-52w5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:02.421147 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.421113    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:02.421147 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.421150    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:02.421423 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.421167    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52w5b\" (UniqueName: \"kubernetes.io/projected/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-kube-api-access-52w5b\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:02.421423 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.421181    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1dafba9-1caf-4355-bfa1-a3b8cea9155e-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:02.694473 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.694403    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="731b8642-8f48-42d2-82b1-10bece738f1b" path="/var/lib/kubelet/pods/731b8642-8f48-42d2-82b1-10bece738f1b/volumes"
Nov 02 02:08:02.694783 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.694770    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90fc1e25-7db0-4438-b278-07a551ac3314" path="/var/lib/kubelet/pods/90fc1e25-7db0-4438-b278-07a551ac3314/volumes"
Nov 02 02:08:02.695154 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.695141    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce33230d-5eac-461b-b766-a40de8dda983" path="/var/lib/kubelet/pods/ce33230d-5eac-461b-b766-a40de8dda983/volumes"
Nov 02 02:08:02.702820 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.702794    5875 generic.go:356] "Generic (PLEG): container finished" podID="d1dafba9-1caf-4355-bfa1-a3b8cea9155e" containerID="969612c799fd017191d24f85bce96313458da87cffb21d5fda1afd3d27105b2c" exitCode=0
Nov 02 02:08:02.702958 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.702842    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"
Nov 02 02:08:02.702958 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.702868    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg" event={"ID":"d1dafba9-1caf-4355-bfa1-a3b8cea9155e","Type":"ContainerDied","Data":"969612c799fd017191d24f85bce96313458da87cffb21d5fda1afd3d27105b2c"}
Nov 02 02:08:02.702958 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.702931    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg" event={"ID":"d1dafba9-1caf-4355-bfa1-a3b8cea9155e","Type":"ContainerDied","Data":"f3f445947fd406fa4d56e6ccff3588b44bf1f96bcc4a8b3a52728cabca976499"}
Nov 02 02:08:02.703109 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.702965    5875 scope.go:122] "RemoveContainer" containerID="969612c799fd017191d24f85bce96313458da87cffb21d5fda1afd3d27105b2c"
Nov 02 02:08:02.713322 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.713280    5875 scope.go:122] "RemoveContainer" containerID="a0089823b223237cf52b7b36d05236b80bdae136acb09f33349f664343d476a6"
Nov 02 02:08:02.722680 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.722664    5875 scope.go:122] "RemoveContainer" containerID="969612c799fd017191d24f85bce96313458da87cffb21d5fda1afd3d27105b2c"
Nov 02 02:08:02.722967 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:08:02.722950    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"969612c799fd017191d24f85bce96313458da87cffb21d5fda1afd3d27105b2c\": container with ID starting with 969612c799fd017191d24f85bce96313458da87cffb21d5fda1afd3d27105b2c not found: ID does not exist" containerID="969612c799fd017191d24f85bce96313458da87cffb21d5fda1afd3d27105b2c"
Nov 02 02:08:02.723026 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.722973    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"969612c799fd017191d24f85bce96313458da87cffb21d5fda1afd3d27105b2c"} err="failed to get container status \"969612c799fd017191d24f85bce96313458da87cffb21d5fda1afd3d27105b2c\": rpc error: code = NotFound desc = could not find container \"969612c799fd017191d24f85bce96313458da87cffb21d5fda1afd3d27105b2c\": container with ID starting with 969612c799fd017191d24f85bce96313458da87cffb21d5fda1afd3d27105b2c not found: ID does not exist"
Nov 02 02:08:02.723026 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.722989    5875 scope.go:122] "RemoveContainer" containerID="a0089823b223237cf52b7b36d05236b80bdae136acb09f33349f664343d476a6"
Nov 02 02:08:02.723335 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:08:02.723215    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0089823b223237cf52b7b36d05236b80bdae136acb09f33349f664343d476a6\": container with ID starting with a0089823b223237cf52b7b36d05236b80bdae136acb09f33349f664343d476a6 not found: ID does not exist" containerID="a0089823b223237cf52b7b36d05236b80bdae136acb09f33349f664343d476a6"
Nov 02 02:08:02.723386 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.723340    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0089823b223237cf52b7b36d05236b80bdae136acb09f33349f664343d476a6"} err="failed to get container status \"a0089823b223237cf52b7b36d05236b80bdae136acb09f33349f664343d476a6\": rpc error: code = NotFound desc = could not find container \"a0089823b223237cf52b7b36d05236b80bdae136acb09f33349f664343d476a6\": container with ID starting with a0089823b223237cf52b7b36d05236b80bdae136acb09f33349f664343d476a6 not found: ID does not exist"
Nov 02 02:08:02.867590 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.867563    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"]
Nov 02 02:08:02.916609 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:02.916584    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c1aad5-openshift-default-6b9469f649-fvbqg"]
Nov 02 02:08:04.694697 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:04.694669    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1dafba9-1caf-4355-bfa1-a3b8cea9155e" path="/var/lib/kubelet/pods/d1dafba9-1caf-4355-bfa1-a3b8cea9155e/volumes"
Nov 02 02:08:05.886212 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:05.886173    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7" podUID="0c003811-1913-436c-b5c2-5e67bc7b72a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:08:07.182444 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.182413    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd"]
Nov 02 02:08:07.183055 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183034    5875 cpu_manager.go:414] "containerMap: removing container" podUID="fda26851-8e30-489a-896d-73d9bcdd783f" containerName="istio-proxy"
Nov 02 02:08:07.183055 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183052    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fda26851-8e30-489a-896d-73d9bcdd783f" containerName="istio-proxy"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183070    5875 cpu_manager.go:414] "containerMap: removing container" podUID="eb67a90f-51c6-4440-923d-db0edea22576" containerName="istio-proxy"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183078    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb67a90f-51c6-4440-923d-db0edea22576" containerName="istio-proxy"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183093    5875 cpu_manager.go:414] "containerMap: removing container" podUID="3986fc99-87a5-41cf-bb18-0815f5e56d96" containerName="maas-api"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183098    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3986fc99-87a5-41cf-bb18-0815f5e56d96" containerName="maas-api"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183104    5875 cpu_manager.go:414] "containerMap: removing container" podUID="d1dafba9-1caf-4355-bfa1-a3b8cea9155e" containerName="istio-proxy"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183108    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1dafba9-1caf-4355-bfa1-a3b8cea9155e" containerName="istio-proxy"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183113    5875 cpu_manager.go:414] "containerMap: removing container" podUID="eb67a90f-51c6-4440-923d-db0edea22576" containerName="istio-proxy"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183117    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb67a90f-51c6-4440-923d-db0edea22576" containerName="istio-proxy"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183127    5875 cpu_manager.go:414] "containerMap: removing container" podUID="bb3138b1-f0f9-4801-9ba7-3ed676116fca" containerName="payload-pre-processing"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183131    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3138b1-f0f9-4801-9ba7-3ed676116fca" containerName="payload-pre-processing"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183138    5875 cpu_manager.go:414] "containerMap: removing container" podUID="fda26851-8e30-489a-896d-73d9bcdd783f" containerName="istio-proxy"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183143    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fda26851-8e30-489a-896d-73d9bcdd783f" containerName="istio-proxy"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183157    5875 cpu_manager.go:414] "containerMap: removing container" podUID="6b93b73a-53d8-463f-8f68-4490e7a8253f" containerName="maas-api"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183164    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b93b73a-53d8-463f-8f68-4490e7a8253f" containerName="maas-api"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183176    5875 cpu_manager.go:414] "containerMap: removing container" podUID="fc421d13-d643-4058-bcfa-1a44a56365eb" containerName="maas-api"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183181    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc421d13-d643-4058-bcfa-1a44a56365eb" containerName="maas-api"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183187    5875 cpu_manager.go:414] "containerMap: removing container" podUID="731b8642-8f48-42d2-82b1-10bece738f1b" containerName="maas-api"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183192    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="731b8642-8f48-42d2-82b1-10bece738f1b" containerName="maas-api"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183199    5875 cpu_manager.go:414] "containerMap: removing container" podUID="8425af9f-79bc-4f19-9109-f6712d7b8198" containerName="payload-processing"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183203    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8425af9f-79bc-4f19-9109-f6712d7b8198" containerName="payload-processing"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183210    5875 cpu_manager.go:414] "containerMap: removing container" podUID="90fc1e25-7db0-4438-b278-07a551ac3314" containerName="payload-pre-processing"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183213    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90fc1e25-7db0-4438-b278-07a551ac3314" containerName="payload-pre-processing"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183232    5875 cpu_manager.go:414] "containerMap: removing container" podUID="a0c5779f-1d48-4749-bcb0-8313b202142d" containerName="istio-proxy"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183239    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c5779f-1d48-4749-bcb0-8313b202142d" containerName="istio-proxy"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183252    5875 cpu_manager.go:414] "containerMap: removing container" podUID="58d02a0f-00d6-4a8f-ac21-b473bd8a440e" containerName="payload-pre-processing"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183259    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58d02a0f-00d6-4a8f-ac21-b473bd8a440e" containerName="payload-pre-processing"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183267    5875 cpu_manager.go:414] "containerMap: removing container" podUID="f6d65404-5fe2-41e1-b878-634a13c47eb8" containerName="payload-processing"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183273    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6d65404-5fe2-41e1-b878-634a13c47eb8" containerName="payload-processing"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183279    5875 cpu_manager.go:414] "containerMap: removing container" podUID="0656b83a-c10d-4929-b029-c3852202b703" containerName="payload-processing"
Nov 02 02:08:07.183269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183283    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0656b83a-c10d-4929-b029-c3852202b703" containerName="payload-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183291    5875 cpu_manager.go:414] "containerMap: removing container" podUID="4eda11d8-9d48-47e9-b94a-223847be8d73" containerName="maas-api"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183295    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eda11d8-9d48-47e9-b94a-223847be8d73" containerName="maas-api"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183302    5875 cpu_manager.go:414] "containerMap: removing container" podUID="d4b7a180-01b7-4c32-93be-dcb96b6b3b42" containerName="istio-proxy"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183307    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4b7a180-01b7-4c32-93be-dcb96b6b3b42" containerName="istio-proxy"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183312    5875 cpu_manager.go:414] "containerMap: removing container" podUID="a587ba49-1db6-49fb-9b21-9a00b9eb504f" containerName="payload-pre-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183317    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a587ba49-1db6-49fb-9b21-9a00b9eb504f" containerName="payload-pre-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183324    5875 cpu_manager.go:414] "containerMap: removing container" podUID="ade83e3d-3ae8-4732-9a85-955c54135ead" containerName="payload-pre-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183328    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade83e3d-3ae8-4732-9a85-955c54135ead" containerName="payload-pre-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183337    5875 cpu_manager.go:414] "containerMap: removing container" podUID="0d212576-5a9c-4523-8a0f-93b851be6406" containerName="payload-pre-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183341    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d212576-5a9c-4523-8a0f-93b851be6406" containerName="payload-pre-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183352    5875 cpu_manager.go:414] "containerMap: removing container" podUID="ce33230d-5eac-461b-b766-a40de8dda983" containerName="payload-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183357    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce33230d-5eac-461b-b766-a40de8dda983" containerName="payload-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183367    5875 cpu_manager.go:414] "containerMap: removing container" podUID="6e39fc3d-d730-4df7-8d10-2376656a88d2" containerName="main"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183371    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e39fc3d-d730-4df7-8d10-2376656a88d2" containerName="main"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183378    5875 cpu_manager.go:414] "containerMap: removing container" podUID="72076bb7-2227-495a-a707-5bfba8979668" containerName="payload-pre-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183383    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72076bb7-2227-495a-a707-5bfba8979668" containerName="payload-pre-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183391    5875 cpu_manager.go:414] "containerMap: removing container" podUID="314d6835-e1c6-48ba-b4eb-a744a699c061" containerName="maas-api"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183395    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="314d6835-e1c6-48ba-b4eb-a744a699c061" containerName="maas-api"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183402    5875 cpu_manager.go:414] "containerMap: removing container" podUID="6c95abfc-9178-4ce3-9e78-cab75960a726" containerName="istio-proxy"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183406    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c95abfc-9178-4ce3-9e78-cab75960a726" containerName="istio-proxy"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183416    5875 cpu_manager.go:414] "containerMap: removing container" podUID="40416082-9ca7-4bb0-91f8-534092c7d0c1" containerName="payload-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183421    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40416082-9ca7-4bb0-91f8-534092c7d0c1" containerName="payload-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183427    5875 cpu_manager.go:414] "containerMap: removing container" podUID="4b5be69e-c305-4530-8322-adf910287bfe" containerName="payload-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183431    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b5be69e-c305-4530-8322-adf910287bfe" containerName="payload-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183440    5875 cpu_manager.go:414] "containerMap: removing container" podUID="99df19ae-03ba-4b87-ae0d-8d837d693d7d" containerName="maas-api"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183445    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99df19ae-03ba-4b87-ae0d-8d837d693d7d" containerName="maas-api"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183453    5875 cpu_manager.go:414] "containerMap: removing container" podUID="d1dafba9-1caf-4355-bfa1-a3b8cea9155e" containerName="istio-proxy"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183458    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1dafba9-1caf-4355-bfa1-a3b8cea9155e" containerName="istio-proxy"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183465    5875 cpu_manager.go:414] "containerMap: removing container" podUID="e52a668a-65c7-4961-9af7-8f176d9f2c2b" containerName="payload-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183469    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52a668a-65c7-4961-9af7-8f176d9f2c2b" containerName="payload-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183477    5875 cpu_manager.go:414] "containerMap: removing container" podUID="160fc571-588a-47fd-9acc-f82010429400" containerName="istio-proxy"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183481    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="160fc571-588a-47fd-9acc-f82010429400" containerName="istio-proxy"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183487    5875 cpu_manager.go:414] "containerMap: removing container" podUID="32eed2d4-07e6-41ba-885c-438ddfe308c7" containerName="payload-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183495    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32eed2d4-07e6-41ba-885c-438ddfe308c7" containerName="payload-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183501    5875 cpu_manager.go:414] "containerMap: removing container" podUID="160fc571-588a-47fd-9acc-f82010429400" containerName="istio-proxy"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183505    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="160fc571-588a-47fd-9acc-f82010429400" containerName="istio-proxy"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183597    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="8425af9f-79bc-4f19-9109-f6712d7b8198" containerName="payload-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183604    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="90fc1e25-7db0-4438-b278-07a551ac3314" containerName="payload-pre-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183609    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c5779f-1d48-4749-bcb0-8313b202142d" containerName="istio-proxy"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183616    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="58d02a0f-00d6-4a8f-ac21-b473bd8a440e" containerName="payload-pre-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183621    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6d65404-5fe2-41e1-b878-634a13c47eb8" containerName="payload-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183626    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="0656b83a-c10d-4929-b029-c3852202b703" containerName="payload-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183631    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eda11d8-9d48-47e9-b94a-223847be8d73" containerName="maas-api"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183635    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4b7a180-01b7-4c32-93be-dcb96b6b3b42" containerName="istio-proxy"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183640    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="a587ba49-1db6-49fb-9b21-9a00b9eb504f" containerName="payload-pre-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183644    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade83e3d-3ae8-4732-9a85-955c54135ead" containerName="payload-pre-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183652    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d212576-5a9c-4523-8a0f-93b851be6406" containerName="payload-pre-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183658    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce33230d-5eac-461b-b766-a40de8dda983" containerName="payload-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183664    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e39fc3d-d730-4df7-8d10-2376656a88d2" containerName="main"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183668    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="72076bb7-2227-495a-a707-5bfba8979668" containerName="payload-pre-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183674    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="314d6835-e1c6-48ba-b4eb-a744a699c061" containerName="maas-api"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183679    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c95abfc-9178-4ce3-9e78-cab75960a726" containerName="istio-proxy"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183684    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4b7a180-01b7-4c32-93be-dcb96b6b3b42" containerName="istio-proxy"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183689    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="40416082-9ca7-4bb0-91f8-534092c7d0c1" containerName="payload-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183695    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b5be69e-c305-4530-8322-adf910287bfe" containerName="payload-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183701    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="99df19ae-03ba-4b87-ae0d-8d837d693d7d" containerName="maas-api"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183706    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c5779f-1d48-4749-bcb0-8313b202142d" containerName="istio-proxy"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183713    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1dafba9-1caf-4355-bfa1-a3b8cea9155e" containerName="istio-proxy"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183718    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c95abfc-9178-4ce3-9e78-cab75960a726" containerName="istio-proxy"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183723    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="e52a668a-65c7-4961-9af7-8f176d9f2c2b" containerName="payload-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183727    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="160fc571-588a-47fd-9acc-f82010429400" containerName="istio-proxy"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183732    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="32eed2d4-07e6-41ba-885c-438ddfe308c7" containerName="payload-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183736    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="160fc571-588a-47fd-9acc-f82010429400" containerName="istio-proxy"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183742    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="fda26851-8e30-489a-896d-73d9bcdd783f" containerName="istio-proxy"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183749    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb67a90f-51c6-4440-923d-db0edea22576" containerName="istio-proxy"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183756    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="3986fc99-87a5-41cf-bb18-0815f5e56d96" containerName="maas-api"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183761    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1dafba9-1caf-4355-bfa1-a3b8cea9155e" containerName="istio-proxy"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183767    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb67a90f-51c6-4440-923d-db0edea22576" containerName="istio-proxy"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183773    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3138b1-f0f9-4801-9ba7-3ed676116fca" containerName="payload-pre-processing"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183778    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="fda26851-8e30-489a-896d-73d9bcdd783f" containerName="istio-proxy"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183784    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b93b73a-53d8-463f-8f68-4490e7a8253f" containerName="maas-api"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183790    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc421d13-d643-4058-bcfa-1a44a56365eb" containerName="maas-api"
Nov 02 02:08:07.184206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.183797    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="731b8642-8f48-42d2-82b1-10bece738f1b" containerName="maas-api"
Nov 02 02:08:07.188742 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.188724    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd"
Nov 02 02:08:07.279143 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.279108    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef1f600f-0137-46f5-bcfa-9357925458e4-maas-api-tls\") pod \"maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd\" (UID: \"ef1f600f-0137-46f5-bcfa-9357925458e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd"
Nov 02 02:08:07.279367 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.279342    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef1f600f-0137-46f5-bcfa-9357925458e4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd\" (UID: \"ef1f600f-0137-46f5-bcfa-9357925458e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd"
Nov 02 02:08:07.279466 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.279444    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef1f600f-0137-46f5-bcfa-9357925458e4-metrics-certs\") pod \"maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd\" (UID: \"ef1f600f-0137-46f5-bcfa-9357925458e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd"
Nov 02 02:08:07.279512 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.279483    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfhnd\" (UniqueName: \"kubernetes.io/projected/ef1f600f-0137-46f5-bcfa-9357925458e4-kube-api-access-mfhnd\") pod \"maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd\" (UID: \"ef1f600f-0137-46f5-bcfa-9357925458e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd"
Nov 02 02:08:07.291409 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.291386    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1c942d-serving-cert\""
Nov 02 02:08:07.294048 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.294026    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd"]
Nov 02 02:08:07.381141 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.381103    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef1f600f-0137-46f5-bcfa-9357925458e4-metrics-certs\") pod \"maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd\" (UID: \"ef1f600f-0137-46f5-bcfa-9357925458e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd"
Nov 02 02:08:07.381141 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.381145    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfhnd\" (UniqueName: \"kubernetes.io/projected/ef1f600f-0137-46f5-bcfa-9357925458e4-kube-api-access-mfhnd\") pod \"maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd\" (UID: \"ef1f600f-0137-46f5-bcfa-9357925458e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd"
Nov 02 02:08:07.381383 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.381256    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef1f600f-0137-46f5-bcfa-9357925458e4-maas-api-tls\") pod \"maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd\" (UID: \"ef1f600f-0137-46f5-bcfa-9357925458e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd"
Nov 02 02:08:07.381383 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.381354    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef1f600f-0137-46f5-bcfa-9357925458e4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd\" (UID: \"ef1f600f-0137-46f5-bcfa-9357925458e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd"
Nov 02 02:08:07.381521 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:08:07.381396    5875 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-serving-cert: secret "maas-api-e2e-inf-a-1c942d-serving-cert" not found
Nov 02 02:08:07.381521 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:08:07.381460    5875 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef1f600f-0137-46f5-bcfa-9357925458e4-maas-api-tls podName:ef1f600f-0137-46f5-bcfa-9357925458e4 nodeName:}" failed. No retries permitted until 2026-11-02 02:08:07.881444904 +0000 UTC m=+2135.322481398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ef1f600f-0137-46f5-bcfa-9357925458e4-maas-api-tls") pod "maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd" (UID: "ef1f600f-0137-46f5-bcfa-9357925458e4") : secret "maas-api-e2e-inf-a-1c942d-serving-cert" not found
Nov 02 02:08:07.381947 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.381929    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef1f600f-0137-46f5-bcfa-9357925458e4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd\" (UID: \"ef1f600f-0137-46f5-bcfa-9357925458e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd"
Nov 02 02:08:07.383599 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.383577    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef1f600f-0137-46f5-bcfa-9357925458e4-metrics-certs\") pod \"maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd\" (UID: \"ef1f600f-0137-46f5-bcfa-9357925458e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd"
Nov 02 02:08:07.448375 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.448083    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfhnd\" (UniqueName: \"kubernetes.io/projected/ef1f600f-0137-46f5-bcfa-9357925458e4-kube-api-access-mfhnd\") pod \"maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd\" (UID: \"ef1f600f-0137-46f5-bcfa-9357925458e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd"
Nov 02 02:08:07.887449 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.887417    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef1f600f-0137-46f5-bcfa-9357925458e4-maas-api-tls\") pod \"maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd\" (UID: \"ef1f600f-0137-46f5-bcfa-9357925458e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd"
Nov 02 02:08:07.890112 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:07.890088    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef1f600f-0137-46f5-bcfa-9357925458e4-maas-api-tls\") pod \"maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd\" (UID: \"ef1f600f-0137-46f5-bcfa-9357925458e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd"
Nov 02 02:08:08.103534 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:08.103493    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd"
Nov 02 02:08:08.296674 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:08.296643    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"]
Nov 02 02:08:08.297143 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:08.296839    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg" podUID="9a054e97-d2a9-47fd-a380-24f4c97d908e" containerName="istio-proxy" containerID="cri-o://aa11d9cba343b58e066c24492392dde2d37937699ea3b8a4c99027184bdf4147" gracePeriod=30
Nov 02 02:08:08.589878 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:08.589839    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd"]
Nov 02 02:08:08.629295 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:08:08.629265    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1f600f_0137_46f5_bcfa_9357925458e4.slice/crio-026dd022d4bd3b71c4d852f84d21993c162c0a80f9e2544b6d976868a2b1b10b WatchSource:0}: Error finding container 026dd022d4bd3b71c4d852f84d21993c162c0a80f9e2544b6d976868a2b1b10b: Status 404 returned error can't find the container with id 026dd022d4bd3b71c4d852f84d21993c162c0a80f9e2544b6d976868a2b1b10b
Nov 02 02:08:08.743010 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:08.742970    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd" event={"ID":"ef1f600f-0137-46f5-bcfa-9357925458e4","Type":"ContainerStarted","Data":"026dd022d4bd3b71c4d852f84d21993c162c0a80f9e2544b6d976868a2b1b10b"}
Nov 02 02:08:09.751901 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:09.751567    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd" event={"ID":"ef1f600f-0137-46f5-bcfa-9357925458e4","Type":"ContainerStarted","Data":"75e5a5f82f7810e6b3b0f1ac778c4e7c47ae38968dcbd7acbfade1502deeb720"}
Nov 02 02:08:09.751901 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:09.751709    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd"
Nov 02 02:08:09.883181 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:09.883125    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd" podStartSLOduration=2.017802115 podStartE2EDuration="2.883111112s" totalImagesPullingTime="865.308997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:08:09.881257603 +0000 UTC m=+2137.322294101" watchObservedRunningTime="2026-11-02 02:08:09.883111112 +0000 UTC m=+2137.324147662"
Nov 02 02:08:10.529669 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:10.529631    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z"]
Nov 02 02:08:10.763928 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:10.763887    5875 patch_prober.go:29] interesting pod/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:08:10.764302 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:10.763938    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg" podUID="9a054e97-d2a9-47fd-a380-24f4c97d908e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:08:11.318613 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:08:11.318572    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:08:12.719272 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:12.719245    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7"
Nov 02 02:08:12.787759 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:12.787722    5875 generic.go:356] "Generic (PLEG): container finished" podID="0c003811-1913-436c-b5c2-5e67bc7b72a1" containerID="4e7d61ecb78d274040844a90e86dc13a0279f791eefcf0d099e0bde3b58bff12" exitCode=0
Nov 02 02:08:12.788193 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:12.787799    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7"
Nov 02 02:08:12.788193 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:12.787817    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7" event={"ID":"0c003811-1913-436c-b5c2-5e67bc7b72a1","Type":"ContainerDied","Data":"4e7d61ecb78d274040844a90e86dc13a0279f791eefcf0d099e0bde3b58bff12"}
Nov 02 02:08:12.788498 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:12.788194    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7" event={"ID":"0c003811-1913-436c-b5c2-5e67bc7b72a1","Type":"ContainerDied","Data":"b4a34bf158f1f959cf0c26eec86d066588c57d91cdf83e53a33ef517745c2481"}
Nov 02 02:08:12.788498 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:12.788257    5875 scope.go:122] "RemoveContainer" containerID="4e7d61ecb78d274040844a90e86dc13a0279f791eefcf0d099e0bde3b58bff12"
Nov 02 02:08:12.801557 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:12.801536    5875 scope.go:122] "RemoveContainer" containerID="4e7d61ecb78d274040844a90e86dc13a0279f791eefcf0d099e0bde3b58bff12"
Nov 02 02:08:12.801852 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:08:12.801830    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e7d61ecb78d274040844a90e86dc13a0279f791eefcf0d099e0bde3b58bff12\": container with ID starting with 4e7d61ecb78d274040844a90e86dc13a0279f791eefcf0d099e0bde3b58bff12 not found: ID does not exist" containerID="4e7d61ecb78d274040844a90e86dc13a0279f791eefcf0d099e0bde3b58bff12"
Nov 02 02:08:12.801899 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:12.801864    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e7d61ecb78d274040844a90e86dc13a0279f791eefcf0d099e0bde3b58bff12"} err="failed to get container status \"4e7d61ecb78d274040844a90e86dc13a0279f791eefcf0d099e0bde3b58bff12\": rpc error: code = NotFound desc = could not find container \"4e7d61ecb78d274040844a90e86dc13a0279f791eefcf0d099e0bde3b58bff12\": container with ID starting with 4e7d61ecb78d274040844a90e86dc13a0279f791eefcf0d099e0bde3b58bff12 not found: ID does not exist"
Nov 02 02:08:12.807393 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:12.807365    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c003811-1913-436c-b5c2-5e67bc7b72a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c003811-1913-436c-b5c2-5e67bc7b72a1-plugins-config-volume\") pod \"0c003811-1913-436c-b5c2-5e67bc7b72a1\" (UID: \"0c003811-1913-436c-b5c2-5e67bc7b72a1\") "
Nov 02 02:08:12.807506 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:12.807439    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c003811-1913-436c-b5c2-5e67bc7b72a1-kube-api-access-gmpg4\" (UniqueName: \"kubernetes.io/projected/0c003811-1913-436c-b5c2-5e67bc7b72a1-kube-api-access-gmpg4\") pod \"0c003811-1913-436c-b5c2-5e67bc7b72a1\" (UID: \"0c003811-1913-436c-b5c2-5e67bc7b72a1\") "
Nov 02 02:08:12.807692 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:12.807673    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c003811-1913-436c-b5c2-5e67bc7b72a1-plugins-config-volume" pod "0c003811-1913-436c-b5c2-5e67bc7b72a1" (UID: "0c003811-1913-436c-b5c2-5e67bc7b72a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:12.807859 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:12.807846    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c003811-1913-436c-b5c2-5e67bc7b72a1-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:12.809676 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:12.809656    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c003811-1913-436c-b5c2-5e67bc7b72a1-kube-api-access-gmpg4" pod "0c003811-1913-436c-b5c2-5e67bc7b72a1" (UID: "0c003811-1913-436c-b5c2-5e67bc7b72a1"). InnerVolumeSpecName "kube-api-access-gmpg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:12.909040 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:12.909006    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmpg4\" (UniqueName: \"kubernetes.io/projected/0c003811-1913-436c-b5c2-5e67bc7b72a1-kube-api-access-gmpg4\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:13.209264 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.209199    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7"]
Nov 02 02:08:13.213275 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.213250    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ed0600-75ff9bdbd9-2btl7"]
Nov 02 02:08:13.574089 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.574066    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 02:08:13.719758 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.719722    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-podinfo\") pod \"9a054e97-d2a9-47fd-a380-24f4c97d908e\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") "
Nov 02 02:08:13.720256 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.719837    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a054e97-d2a9-47fd-a380-24f4c97d908e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a054e97-d2a9-47fd-a380-24f4c97d908e-istiod-ca-cert\") pod \"9a054e97-d2a9-47fd-a380-24f4c97d908e\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") "
Nov 02 02:08:13.720256 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.719887    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-envoy\") pod \"9a054e97-d2a9-47fd-a380-24f4c97d908e\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") "
Nov 02 02:08:13.720256 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.719904    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-credential-socket\") pod \"9a054e97-d2a9-47fd-a380-24f4c97d908e\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") "
Nov 02 02:08:13.720256 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.719931    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-workload-socket\") pod \"9a054e97-d2a9-47fd-a380-24f4c97d908e\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") "
Nov 02 02:08:13.720256 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.719959    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-data\") pod \"9a054e97-d2a9-47fd-a380-24f4c97d908e\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") "
Nov 02 02:08:13.720256 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.719988    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-workload-certs\") pod \"9a054e97-d2a9-47fd-a380-24f4c97d908e\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") "
Nov 02 02:08:13.720256 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.720018    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-token\" (UniqueName: \"kubernetes.io/projected/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-token\") pod \"9a054e97-d2a9-47fd-a380-24f4c97d908e\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") "
Nov 02 02:08:13.720256 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.720115    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a054e97-d2a9-47fd-a380-24f4c97d908e-kube-api-access-wsm2p\" (UniqueName: \"kubernetes.io/projected/9a054e97-d2a9-47fd-a380-24f4c97d908e-kube-api-access-wsm2p\") pod \"9a054e97-d2a9-47fd-a380-24f4c97d908e\" (UID: \"9a054e97-d2a9-47fd-a380-24f4c97d908e\") "
Nov 02 02:08:13.720668 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.720364    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a054e97-d2a9-47fd-a380-24f4c97d908e-istiod-ca-cert" pod "9a054e97-d2a9-47fd-a380-24f4c97d908e" (UID: "9a054e97-d2a9-47fd-a380-24f4c97d908e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:13.720668 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.720440    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-workload-socket" pod "9a054e97-d2a9-47fd-a380-24f4c97d908e" (UID: "9a054e97-d2a9-47fd-a380-24f4c97d908e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:13.720668 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.720564    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-workload-certs" pod "9a054e97-d2a9-47fd-a380-24f4c97d908e" (UID: "9a054e97-d2a9-47fd-a380-24f4c97d908e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:13.720668 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.720611    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-data" pod "9a054e97-d2a9-47fd-a380-24f4c97d908e" (UID: "9a054e97-d2a9-47fd-a380-24f4c97d908e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:13.720856 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.720806    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a054e97-d2a9-47fd-a380-24f4c97d908e-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:13.720856 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.720818    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:13.720856 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.720828    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:13.720856 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.720837    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:13.721239 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.720881    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-credential-socket" pod "9a054e97-d2a9-47fd-a380-24f4c97d908e" (UID: "9a054e97-d2a9-47fd-a380-24f4c97d908e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:13.722426 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.722389    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-envoy" pod "9a054e97-d2a9-47fd-a380-24f4c97d908e" (UID: "9a054e97-d2a9-47fd-a380-24f4c97d908e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:13.722628 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.722611    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-token" pod "9a054e97-d2a9-47fd-a380-24f4c97d908e" (UID: "9a054e97-d2a9-47fd-a380-24f4c97d908e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:13.722791 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.722778    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a054e97-d2a9-47fd-a380-24f4c97d908e-kube-api-access-wsm2p" pod "9a054e97-d2a9-47fd-a380-24f4c97d908e" (UID: "9a054e97-d2a9-47fd-a380-24f4c97d908e"). InnerVolumeSpecName "kube-api-access-wsm2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:13.723018 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.722991    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-podinfo" pod "9a054e97-d2a9-47fd-a380-24f4c97d908e" (UID: "9a054e97-d2a9-47fd-a380-24f4c97d908e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:08:13.796414 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.796382    5875 generic.go:356] "Generic (PLEG): container finished" podID="9a054e97-d2a9-47fd-a380-24f4c97d908e" containerID="aa11d9cba343b58e066c24492392dde2d37937699ea3b8a4c99027184bdf4147" exitCode=0
Nov 02 02:08:13.796597 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.796431    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"
Nov 02 02:08:13.796597 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.796473    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg" event={"ID":"9a054e97-d2a9-47fd-a380-24f4c97d908e","Type":"ContainerDied","Data":"aa11d9cba343b58e066c24492392dde2d37937699ea3b8a4c99027184bdf4147"}
Nov 02 02:08:13.796597 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.796524    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg" event={"ID":"9a054e97-d2a9-47fd-a380-24f4c97d908e","Type":"ContainerDied","Data":"0c14353e823619ae7e46250b799a631bf6dbcc75938f5021019dcd300f5a1622"}
Nov 02 02:08:13.796597 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.796549    5875 scope.go:122] "RemoveContainer" containerID="aa11d9cba343b58e066c24492392dde2d37937699ea3b8a4c99027184bdf4147"
Nov 02 02:08:13.807180 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.807166    5875 scope.go:122] "RemoveContainer" containerID="19244c9bd784f673fa14b8bef85d37ceeefef9c38799461dfde948ae773e2ee0"
Nov 02 02:08:13.816277 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.816257    5875 scope.go:122] "RemoveContainer" containerID="aa11d9cba343b58e066c24492392dde2d37937699ea3b8a4c99027184bdf4147"
Nov 02 02:08:13.816551 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:08:13.816533    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa11d9cba343b58e066c24492392dde2d37937699ea3b8a4c99027184bdf4147\": container with ID starting with aa11d9cba343b58e066c24492392dde2d37937699ea3b8a4c99027184bdf4147 not found: ID does not exist" containerID="aa11d9cba343b58e066c24492392dde2d37937699ea3b8a4c99027184bdf4147"
Nov 02 02:08:13.816612 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.816558    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa11d9cba343b58e066c24492392dde2d37937699ea3b8a4c99027184bdf4147"} err="failed to get container status \"aa11d9cba343b58e066c24492392dde2d37937699ea3b8a4c99027184bdf4147\": rpc error: code = NotFound desc = could not find container \"aa11d9cba343b58e066c24492392dde2d37937699ea3b8a4c99027184bdf4147\": container with ID starting with aa11d9cba343b58e066c24492392dde2d37937699ea3b8a4c99027184bdf4147 not found: ID does not exist"
Nov 02 02:08:13.816612 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.816578    5875 scope.go:122] "RemoveContainer" containerID="19244c9bd784f673fa14b8bef85d37ceeefef9c38799461dfde948ae773e2ee0"
Nov 02 02:08:13.816815 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:08:13.816797    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19244c9bd784f673fa14b8bef85d37ceeefef9c38799461dfde948ae773e2ee0\": container with ID starting with 19244c9bd784f673fa14b8bef85d37ceeefef9c38799461dfde948ae773e2ee0 not found: ID does not exist" containerID="19244c9bd784f673fa14b8bef85d37ceeefef9c38799461dfde948ae773e2ee0"
Nov 02 02:08:13.816854 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.816821    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19244c9bd784f673fa14b8bef85d37ceeefef9c38799461dfde948ae773e2ee0"} err="failed to get container status \"19244c9bd784f673fa14b8bef85d37ceeefef9c38799461dfde948ae773e2ee0\": rpc error: code = NotFound desc = could not find container \"19244c9bd784f673fa14b8bef85d37ceeefef9c38799461dfde948ae773e2ee0\": container with ID starting with 19244c9bd784f673fa14b8bef85d37ceeefef9c38799461dfde948ae773e2ee0 not found: ID does not exist"
Nov 02 02:08:13.821482 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.821464    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsm2p\" (UniqueName: \"kubernetes.io/projected/9a054e97-d2a9-47fd-a380-24f4c97d908e-kube-api-access-wsm2p\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:13.821542 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.821485    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:13.821542 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.821496    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:13.821542 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.821506    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a054e97-d2a9-47fd-a380-24f4c97d908e-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:13.821542 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.821514    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a054e97-d2a9-47fd-a380-24f4c97d908e-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:13.961320 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.961267    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"]
Nov 02 02:08:13.965050 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:13.965017    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-606ca0-openshift-default-69c787cb65-d62hg"]
Nov 02 02:08:14.694707 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:14.694668    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c003811-1913-436c-b5c2-5e67bc7b72a1" path="/var/lib/kubelet/pods/0c003811-1913-436c-b5c2-5e67bc7b72a1/volumes"
Nov 02 02:08:14.695282 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:14.695235    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a054e97-d2a9-47fd-a380-24f4c97d908e" path="/var/lib/kubelet/pods/9a054e97-d2a9-47fd-a380-24f4c97d908e/volumes"
Nov 02 02:08:15.530994 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:15.530837    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z" podUID="32512901-d0e2-4df1-bab5-571d7434ed06" containerName="payload-pre-processing" containerID="cri-o://84cd707cee34835bf493bed0395d3da93a340122bb2ca137f21b675dfb5da182" gracePeriod=25
Nov 02 02:08:15.764117 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:15.764090    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd"
Nov 02 02:08:15.810418 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:15.810395    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z"
Nov 02 02:08:15.812207 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:15.812179    5875 generic.go:356] "Generic (PLEG): container finished" podID="32512901-d0e2-4df1-bab5-571d7434ed06" containerID="84cd707cee34835bf493bed0395d3da93a340122bb2ca137f21b675dfb5da182" exitCode=0
Nov 02 02:08:15.812327 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:15.812213    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z"
Nov 02 02:08:15.812327 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:15.812266    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z" event={"ID":"32512901-d0e2-4df1-bab5-571d7434ed06","Type":"ContainerDied","Data":"84cd707cee34835bf493bed0395d3da93a340122bb2ca137f21b675dfb5da182"}
Nov 02 02:08:15.812400 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:15.812340    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z" event={"ID":"32512901-d0e2-4df1-bab5-571d7434ed06","Type":"ContainerDied","Data":"44a131c0fa7357c94892a14daefc608e292597d037602d697c979451b1030934"}
Nov 02 02:08:15.812400 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:15.812362    5875 scope.go:122] "RemoveContainer" containerID="84cd707cee34835bf493bed0395d3da93a340122bb2ca137f21b675dfb5da182"
Nov 02 02:08:15.822383 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:15.822367    5875 scope.go:122] "RemoveContainer" containerID="84cd707cee34835bf493bed0395d3da93a340122bb2ca137f21b675dfb5da182"
Nov 02 02:08:15.822658 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:08:15.822641    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84cd707cee34835bf493bed0395d3da93a340122bb2ca137f21b675dfb5da182\": container with ID starting with 84cd707cee34835bf493bed0395d3da93a340122bb2ca137f21b675dfb5da182 not found: ID does not exist" containerID="84cd707cee34835bf493bed0395d3da93a340122bb2ca137f21b675dfb5da182"
Nov 02 02:08:15.822700 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:15.822666    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84cd707cee34835bf493bed0395d3da93a340122bb2ca137f21b675dfb5da182"} err="failed to get container status \"84cd707cee34835bf493bed0395d3da93a340122bb2ca137f21b675dfb5da182\": rpc error: code = NotFound desc = could not find container \"84cd707cee34835bf493bed0395d3da93a340122bb2ca137f21b675dfb5da182\": container with ID starting with 84cd707cee34835bf493bed0395d3da93a340122bb2ca137f21b675dfb5da182 not found: ID does not exist"
Nov 02 02:08:15.946079 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:15.946048    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32512901-d0e2-4df1-bab5-571d7434ed06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32512901-d0e2-4df1-bab5-571d7434ed06-plugins-config-volume\") pod \"32512901-d0e2-4df1-bab5-571d7434ed06\" (UID: \"32512901-d0e2-4df1-bab5-571d7434ed06\") "
Nov 02 02:08:15.946246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:15.946187    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32512901-d0e2-4df1-bab5-571d7434ed06-kube-api-access-rr766\" (UniqueName: \"kubernetes.io/projected/32512901-d0e2-4df1-bab5-571d7434ed06-kube-api-access-rr766\") pod \"32512901-d0e2-4df1-bab5-571d7434ed06\" (UID: \"32512901-d0e2-4df1-bab5-571d7434ed06\") "
Nov 02 02:08:15.946445 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:15.946414    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32512901-d0e2-4df1-bab5-571d7434ed06-plugins-config-volume" pod "32512901-d0e2-4df1-bab5-571d7434ed06" (UID: "32512901-d0e2-4df1-bab5-571d7434ed06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:15.946578 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:15.946563    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32512901-d0e2-4df1-bab5-571d7434ed06-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:15.948598 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:15.948574    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32512901-d0e2-4df1-bab5-571d7434ed06-kube-api-access-rr766" pod "32512901-d0e2-4df1-bab5-571d7434ed06" (UID: "32512901-d0e2-4df1-bab5-571d7434ed06"). InnerVolumeSpecName "kube-api-access-rr766". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:16.048158 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:16.048084    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr766\" (UniqueName: \"kubernetes.io/projected/32512901-d0e2-4df1-bab5-571d7434ed06-kube-api-access-rr766\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:16.232547 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:16.232514    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z"]
Nov 02 02:08:16.236405 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:16.236380    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d1405a-6f647b579f-b6j4z"]
Nov 02 02:08:16.698261 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:16.697338    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32512901-d0e2-4df1-bab5-571d7434ed06" path="/var/lib/kubelet/pods/32512901-d0e2-4df1-bab5-571d7434ed06/volumes"
Nov 02 02:08:16.964975 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:16.964947    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m"]
Nov 02 02:08:16.965554 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:16.965539    5875 cpu_manager.go:414] "containerMap: removing container" podUID="32512901-d0e2-4df1-bab5-571d7434ed06" containerName="payload-pre-processing"
Nov 02 02:08:16.965554 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:16.965554    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32512901-d0e2-4df1-bab5-571d7434ed06" containerName="payload-pre-processing"
Nov 02 02:08:16.965678 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:16.965568    5875 cpu_manager.go:414] "containerMap: removing container" podUID="6c95abfc-9178-4ce3-9e78-cab75960a726" containerName="istio-proxy"
Nov 02 02:08:16.965678 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:16.965573    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c95abfc-9178-4ce3-9e78-cab75960a726" containerName="istio-proxy"
Nov 02 02:08:16.965678 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:16.965585    5875 cpu_manager.go:414] "containerMap: removing container" podUID="9a054e97-d2a9-47fd-a380-24f4c97d908e" containerName="istio-proxy"
Nov 02 02:08:16.965678 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:16.965589    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a054e97-d2a9-47fd-a380-24f4c97d908e" containerName="istio-proxy"
Nov 02 02:08:16.965678 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:16.965625    5875 cpu_manager.go:414] "containerMap: removing container" podUID="a0c5779f-1d48-4749-bcb0-8313b202142d" containerName="istio-proxy"
Nov 02 02:08:16.965678 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:16.965630    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c5779f-1d48-4749-bcb0-8313b202142d" containerName="istio-proxy"
Nov 02 02:08:16.965678 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:16.965637    5875 cpu_manager.go:414] "containerMap: removing container" podUID="0c003811-1913-436c-b5c2-5e67bc7b72a1" containerName="payload-processing"
Nov 02 02:08:16.965678 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:16.965642    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c003811-1913-436c-b5c2-5e67bc7b72a1" containerName="payload-processing"
Nov 02 02:08:16.965678 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:16.965650    5875 cpu_manager.go:414] "containerMap: removing container" podUID="d4b7a180-01b7-4c32-93be-dcb96b6b3b42" containerName="istio-proxy"
Nov 02 02:08:16.965678 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:16.965654    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4b7a180-01b7-4c32-93be-dcb96b6b3b42" containerName="istio-proxy"
Nov 02 02:08:16.966033 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:16.965730    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a054e97-d2a9-47fd-a380-24f4c97d908e" containerName="istio-proxy"
Nov 02 02:08:16.966033 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:16.965741    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c003811-1913-436c-b5c2-5e67bc7b72a1" containerName="payload-processing"
Nov 02 02:08:16.966033 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:16.965749    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="32512901-d0e2-4df1-bab5-571d7434ed06" containerName="payload-pre-processing"
Nov 02 02:08:16.966033 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:16.965757    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a054e97-d2a9-47fd-a380-24f4c97d908e" containerName="istio-proxy"
Nov 02 02:08:16.970323 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:16.970302    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m"
Nov 02 02:08:17.024179 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.023873    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5700d3d1\""
Nov 02 02:08:17.024179 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.023962    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5700d3d1-dockercfg-tmh9r\""
Nov 02 02:08:17.026811 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.026788    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m"]
Nov 02 02:08:17.075248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.071846    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b440959-cb4b-4694-b8f2-55a98b1f0e06-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m\" (UID: \"7b440959-cb4b-4694-b8f2-55a98b1f0e06\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m"
Nov 02 02:08:17.075248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.071981    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9rt8\" (UniqueName: \"kubernetes.io/projected/7b440959-cb4b-4694-b8f2-55a98b1f0e06-kube-api-access-g9rt8\") pod \"payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m\" (UID: \"7b440959-cb4b-4694-b8f2-55a98b1f0e06\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m"
Nov 02 02:08:17.148088 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.147612    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4"]
Nov 02 02:08:17.148621 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.148598    5875 cpu_manager.go:414] "containerMap: removing container" podUID="9a054e97-d2a9-47fd-a380-24f4c97d908e" containerName="istio-proxy"
Nov 02 02:08:17.148755 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.148623    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a054e97-d2a9-47fd-a380-24f4c97d908e" containerName="istio-proxy"
Nov 02 02:08:17.153352 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.153325    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4"
Nov 02 02:08:17.173078 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.173050    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b440959-cb4b-4694-b8f2-55a98b1f0e06-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m\" (UID: \"7b440959-cb4b-4694-b8f2-55a98b1f0e06\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m"
Nov 02 02:08:17.173247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.173155    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9rt8\" (UniqueName: \"kubernetes.io/projected/7b440959-cb4b-4694-b8f2-55a98b1f0e06-kube-api-access-g9rt8\") pod \"payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m\" (UID: \"7b440959-cb4b-4694-b8f2-55a98b1f0e06\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m"
Nov 02 02:08:17.173734 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.173710    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b440959-cb4b-4694-b8f2-55a98b1f0e06-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m\" (UID: \"7b440959-cb4b-4694-b8f2-55a98b1f0e06\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m"
Nov 02 02:08:17.208721 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.208695    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4"]
Nov 02 02:08:17.234400 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.234334    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9rt8\" (UniqueName: \"kubernetes.io/projected/7b440959-cb4b-4694-b8f2-55a98b1f0e06-kube-api-access-g9rt8\") pod \"payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m\" (UID: \"7b440959-cb4b-4694-b8f2-55a98b1f0e06\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m"
Nov 02 02:08:17.274501 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.274471    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4780a97d-8456-4233-83ad-277e4d864125-plugins-config-volume\") pod \"payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4\" (UID: \"4780a97d-8456-4233-83ad-277e4d864125\") " pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4"
Nov 02 02:08:17.274642 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.274537    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn7cm\" (UniqueName: \"kubernetes.io/projected/4780a97d-8456-4233-83ad-277e4d864125-kube-api-access-qn7cm\") pod \"payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4\" (UID: \"4780a97d-8456-4233-83ad-277e4d864125\") " pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4"
Nov 02 02:08:17.283571 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.283548    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m"
Nov 02 02:08:17.375895 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.375858    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4780a97d-8456-4233-83ad-277e4d864125-plugins-config-volume\") pod \"payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4\" (UID: \"4780a97d-8456-4233-83ad-277e4d864125\") " pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4"
Nov 02 02:08:17.376079 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.375958    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7cm\" (UniqueName: \"kubernetes.io/projected/4780a97d-8456-4233-83ad-277e4d864125-kube-api-access-qn7cm\") pod \"payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4\" (UID: \"4780a97d-8456-4233-83ad-277e4d864125\") " pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4"
Nov 02 02:08:17.377134 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.377065    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4780a97d-8456-4233-83ad-277e4d864125-plugins-config-volume\") pod \"payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4\" (UID: \"4780a97d-8456-4233-83ad-277e4d864125\") " pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4"
Nov 02 02:08:17.436618 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.436584    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn7cm\" (UniqueName: \"kubernetes.io/projected/4780a97d-8456-4233-83ad-277e4d864125-kube-api-access-qn7cm\") pod \"payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4\" (UID: \"4780a97d-8456-4233-83ad-277e4d864125\") " pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4"
Nov 02 02:08:17.472814 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.472788    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4"
Nov 02 02:08:17.679346 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.679313    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m"]
Nov 02 02:08:17.681874 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:08:17.681844    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b440959_cb4b_4694_b8f2_55a98b1f0e06.slice/crio-a4474cfa65c5ba863e85abca73d156f9c27d831f6a8f271ab97ba19db600d9ee WatchSource:0}: Error finding container a4474cfa65c5ba863e85abca73d156f9c27d831f6a8f271ab97ba19db600d9ee: Status 404 returned error can't find the container with id a4474cfa65c5ba863e85abca73d156f9c27d831f6a8f271ab97ba19db600d9ee
Nov 02 02:08:17.827444 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.827407    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m" event={"ID":"7b440959-cb4b-4694-b8f2-55a98b1f0e06","Type":"ContainerStarted","Data":"9de8bbc71d6053b0dcc27387e8d3ce450e0fee2218c70e3c8cb40358d1ed1a17"}
Nov 02 02:08:17.827921 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.827465    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m" event={"ID":"7b440959-cb4b-4694-b8f2-55a98b1f0e06","Type":"ContainerStarted","Data":"a4474cfa65c5ba863e85abca73d156f9c27d831f6a8f271ab97ba19db600d9ee"}
Nov 02 02:08:17.827921 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.827492    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m"
Nov 02 02:08:17.871421 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.871394    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4"]
Nov 02 02:08:17.911843 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:08:17.911813    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4780a97d_8456_4233_83ad_277e4d864125.slice/crio-c990ddd8510d3df2217088bb108718b3b45bdd31f3f8d378203e9a0cc6e0742c WatchSource:0}: Error finding container c990ddd8510d3df2217088bb108718b3b45bdd31f3f8d378203e9a0cc6e0742c: Status 404 returned error can't find the container with id c990ddd8510d3df2217088bb108718b3b45bdd31f3f8d378203e9a0cc6e0742c
Nov 02 02:08:17.944672 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:17.944618    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m" podStartSLOduration=1.944604407 podStartE2EDuration="1.944604407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:08:17.941025213 +0000 UTC m=+2145.382061733" watchObservedRunningTime="2026-11-02 02:08:17.944604407 +0000 UTC m=+2145.385640907"
Nov 02 02:08:18.835672 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:18.835634    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4" event={"ID":"4780a97d-8456-4233-83ad-277e4d864125","Type":"ContainerStarted","Data":"5808a9c710ac47f77a05c6145c24fed3be8114f5a6d99c42de65a076e8424b98"}
Nov 02 02:08:18.836108 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:18.835702    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4" event={"ID":"4780a97d-8456-4233-83ad-277e4d864125","Type":"ContainerStarted","Data":"c990ddd8510d3df2217088bb108718b3b45bdd31f3f8d378203e9a0cc6e0742c"}
Nov 02 02:08:18.836108 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:18.835809    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4"
Nov 02 02:08:18.953402 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:18.953331    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4" podStartSLOduration=1.953316872 podStartE2EDuration="1.953316872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:08:18.950776678 +0000 UTC m=+2146.391813177" watchObservedRunningTime="2026-11-02 02:08:18.953316872 +0000 UTC m=+2146.394353372"
Nov 02 02:08:19.997357 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:19.997319    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq"]
Nov 02 02:08:20.109376 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:20.109343    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q"]
Nov 02 02:08:24.997668 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:24.997628    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq" podUID="409169ea-5827-4946-9836-638b0bd62614" containerName="payload-processing" containerID="cri-o://b973915c53c650330e0b9874b549b911489ba431a25c9dfbcc54259f8f987309" gracePeriod=25
Nov 02 02:08:25.109943 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.109891    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q" podUID="43d6edfb-c755-4afd-9237-1d4ce70352f1" containerName="payload-pre-processing" containerID="cri-o://6383bcca0e7e62e5acab828e8ba43df2438eca88f6ee7b7aeeed7e54649c4b6c" gracePeriod=25
Nov 02 02:08:25.266254 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.266169    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq"
Nov 02 02:08:25.308999 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.308961    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/409169ea-5827-4946-9836-638b0bd62614-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/409169ea-5827-4946-9836-638b0bd62614-plugins-config-volume\") pod \"409169ea-5827-4946-9836-638b0bd62614\" (UID: \"409169ea-5827-4946-9836-638b0bd62614\") "
Nov 02 02:08:25.309142 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.309128    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/409169ea-5827-4946-9836-638b0bd62614-kube-api-access-96x6h\" (UniqueName: \"kubernetes.io/projected/409169ea-5827-4946-9836-638b0bd62614-kube-api-access-96x6h\") pod \"409169ea-5827-4946-9836-638b0bd62614\" (UID: \"409169ea-5827-4946-9836-638b0bd62614\") "
Nov 02 02:08:25.309358 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.309334    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/409169ea-5827-4946-9836-638b0bd62614-plugins-config-volume" pod "409169ea-5827-4946-9836-638b0bd62614" (UID: "409169ea-5827-4946-9836-638b0bd62614"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:25.309569 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.309550    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/409169ea-5827-4946-9836-638b0bd62614-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:25.312427 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.312405    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409169ea-5827-4946-9836-638b0bd62614-kube-api-access-96x6h" pod "409169ea-5827-4946-9836-638b0bd62614" (UID: "409169ea-5827-4946-9836-638b0bd62614"). InnerVolumeSpecName "kube-api-access-96x6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:25.373114 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.373091    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q"
Nov 02 02:08:25.410039 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.410009    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43d6edfb-c755-4afd-9237-1d4ce70352f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d6edfb-c755-4afd-9237-1d4ce70352f1-plugins-config-volume\") pod \"43d6edfb-c755-4afd-9237-1d4ce70352f1\" (UID: \"43d6edfb-c755-4afd-9237-1d4ce70352f1\") "
Nov 02 02:08:25.410258 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.410099    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43d6edfb-c755-4afd-9237-1d4ce70352f1-kube-api-access-9595v\" (UniqueName: \"kubernetes.io/projected/43d6edfb-c755-4afd-9237-1d4ce70352f1-kube-api-access-9595v\") pod \"43d6edfb-c755-4afd-9237-1d4ce70352f1\" (UID: \"43d6edfb-c755-4afd-9237-1d4ce70352f1\") "
Nov 02 02:08:25.410459 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.410435    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d6edfb-c755-4afd-9237-1d4ce70352f1-plugins-config-volume" pod "43d6edfb-c755-4afd-9237-1d4ce70352f1" (UID: "43d6edfb-c755-4afd-9237-1d4ce70352f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:25.410655 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.410638    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96x6h\" (UniqueName: \"kubernetes.io/projected/409169ea-5827-4946-9836-638b0bd62614-kube-api-access-96x6h\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:25.410713 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.410662    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d6edfb-c755-4afd-9237-1d4ce70352f1-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:25.412353 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.412332    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d6edfb-c755-4afd-9237-1d4ce70352f1-kube-api-access-9595v" pod "43d6edfb-c755-4afd-9237-1d4ce70352f1" (UID: "43d6edfb-c755-4afd-9237-1d4ce70352f1"). InnerVolumeSpecName "kube-api-access-9595v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:25.512094 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.512057    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9595v\" (UniqueName: \"kubernetes.io/projected/43d6edfb-c755-4afd-9237-1d4ce70352f1-kube-api-access-9595v\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:25.878525 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.878493    5875 generic.go:356] "Generic (PLEG): container finished" podID="409169ea-5827-4946-9836-638b0bd62614" containerID="b973915c53c650330e0b9874b549b911489ba431a25c9dfbcc54259f8f987309" exitCode=0
Nov 02 02:08:25.878812 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.878541    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq"
Nov 02 02:08:25.878812 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.878580    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq" event={"ID":"409169ea-5827-4946-9836-638b0bd62614","Type":"ContainerDied","Data":"b973915c53c650330e0b9874b549b911489ba431a25c9dfbcc54259f8f987309"}
Nov 02 02:08:25.878812 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.878649    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq" event={"ID":"409169ea-5827-4946-9836-638b0bd62614","Type":"ContainerDied","Data":"368f7c4339fd837fd24bc0d8eb59febf85d70ce15a0ca3e3c3c05c0e6e65887e"}
Nov 02 02:08:25.878812 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.878682    5875 scope.go:122] "RemoveContainer" containerID="b973915c53c650330e0b9874b549b911489ba431a25c9dfbcc54259f8f987309"
Nov 02 02:08:25.880058 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.880033    5875 generic.go:356] "Generic (PLEG): container finished" podID="43d6edfb-c755-4afd-9237-1d4ce70352f1" containerID="6383bcca0e7e62e5acab828e8ba43df2438eca88f6ee7b7aeeed7e54649c4b6c" exitCode=0
Nov 02 02:08:25.880158 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.880078    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q"
Nov 02 02:08:25.880360 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.880232    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q" event={"ID":"43d6edfb-c755-4afd-9237-1d4ce70352f1","Type":"ContainerDied","Data":"6383bcca0e7e62e5acab828e8ba43df2438eca88f6ee7b7aeeed7e54649c4b6c"}
Nov 02 02:08:25.880421 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.880391    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q" event={"ID":"43d6edfb-c755-4afd-9237-1d4ce70352f1","Type":"ContainerDied","Data":"269fd860842e703c2597239122b0d21d65fc8f1376adaf2fe5b79dc09efa4a53"}
Nov 02 02:08:25.890593 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.890576    5875 scope.go:122] "RemoveContainer" containerID="b973915c53c650330e0b9874b549b911489ba431a25c9dfbcc54259f8f987309"
Nov 02 02:08:25.890911 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:08:25.890890    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b973915c53c650330e0b9874b549b911489ba431a25c9dfbcc54259f8f987309\": container with ID starting with b973915c53c650330e0b9874b549b911489ba431a25c9dfbcc54259f8f987309 not found: ID does not exist" containerID="b973915c53c650330e0b9874b549b911489ba431a25c9dfbcc54259f8f987309"
Nov 02 02:08:25.890991 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.890918    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b973915c53c650330e0b9874b549b911489ba431a25c9dfbcc54259f8f987309"} err="failed to get container status \"b973915c53c650330e0b9874b549b911489ba431a25c9dfbcc54259f8f987309\": rpc error: code = NotFound desc = could not find container \"b973915c53c650330e0b9874b549b911489ba431a25c9dfbcc54259f8f987309\": container with ID starting with b973915c53c650330e0b9874b549b911489ba431a25c9dfbcc54259f8f987309 not found: ID does not exist"
Nov 02 02:08:25.890991 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.890934    5875 scope.go:122] "RemoveContainer" containerID="6383bcca0e7e62e5acab828e8ba43df2438eca88f6ee7b7aeeed7e54649c4b6c"
Nov 02 02:08:25.900465 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.900445    5875 scope.go:122] "RemoveContainer" containerID="6383bcca0e7e62e5acab828e8ba43df2438eca88f6ee7b7aeeed7e54649c4b6c"
Nov 02 02:08:25.900745 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:08:25.900726    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6383bcca0e7e62e5acab828e8ba43df2438eca88f6ee7b7aeeed7e54649c4b6c\": container with ID starting with 6383bcca0e7e62e5acab828e8ba43df2438eca88f6ee7b7aeeed7e54649c4b6c not found: ID does not exist" containerID="6383bcca0e7e62e5acab828e8ba43df2438eca88f6ee7b7aeeed7e54649c4b6c"
Nov 02 02:08:25.900815 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:25.900752    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6383bcca0e7e62e5acab828e8ba43df2438eca88f6ee7b7aeeed7e54649c4b6c"} err="failed to get container status \"6383bcca0e7e62e5acab828e8ba43df2438eca88f6ee7b7aeeed7e54649c4b6c\": rpc error: code = NotFound desc = could not find container \"6383bcca0e7e62e5acab828e8ba43df2438eca88f6ee7b7aeeed7e54649c4b6c\": container with ID starting with 6383bcca0e7e62e5acab828e8ba43df2438eca88f6ee7b7aeeed7e54649c4b6c not found: ID does not exist"
Nov 02 02:08:26.043745 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:26.043710    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q"]
Nov 02 02:08:26.093049 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:26.093012    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11f26c-955dc65f9-t565q"]
Nov 02 02:08:26.158510 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:26.158424    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq"]
Nov 02 02:08:26.161881 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:26.161857    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-11f26c-5f48c7bb7-xg8dq"]
Nov 02 02:08:26.696084 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:26.696051    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="409169ea-5827-4946-9836-638b0bd62614" path="/var/lib/kubelet/pods/409169ea-5827-4946-9836-638b0bd62614/volumes"
Nov 02 02:08:26.696501 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:26.696486    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43d6edfb-c755-4afd-9237-1d4ce70352f1" path="/var/lib/kubelet/pods/43d6edfb-c755-4afd-9237-1d4ce70352f1/volumes"
Nov 02 02:08:28.838773 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:28.838744    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m"
Nov 02 02:08:29.845114 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:29.845086    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4"
Nov 02 02:08:33.179512 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:33.179480    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:08:41.318615 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:08:41.318523    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:08:42.242938 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:42.242902    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd"]
Nov 02 02:08:42.243127 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:42.243107    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd" podUID="a5c9f74d-7015-401f-900b-d065938f4507" containerName="maas-api" containerID="cri-o://b896f0500cc649a6f494f98756c4e1f88726f2b8f64db991e95ea5998bde6093" gracePeriod=30
Nov 02 02:08:42.535821 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:42.535798    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd"
Nov 02 02:08:42.648247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:42.632213    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5c9f74d-7015-401f-900b-d065938f4507-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5c9f74d-7015-401f-900b-d065938f4507-metrics-certs\") pod \"a5c9f74d-7015-401f-900b-d065938f4507\" (UID: \"a5c9f74d-7015-401f-900b-d065938f4507\") "
Nov 02 02:08:42.648247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:42.632361    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5c9f74d-7015-401f-900b-d065938f4507-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5c9f74d-7015-401f-900b-d065938f4507-openshift-service-ca\") pod \"a5c9f74d-7015-401f-900b-d065938f4507\" (UID: \"a5c9f74d-7015-401f-900b-d065938f4507\") "
Nov 02 02:08:42.648247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:42.632391    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5c9f74d-7015-401f-900b-d065938f4507-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5c9f74d-7015-401f-900b-d065938f4507-maas-api-tls\") pod \"a5c9f74d-7015-401f-900b-d065938f4507\" (UID: \"a5c9f74d-7015-401f-900b-d065938f4507\") "
Nov 02 02:08:42.648247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:42.632527    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c9f74d-7015-401f-900b-d065938f4507-kube-api-access-m7ghr\" (UniqueName: \"kubernetes.io/projected/a5c9f74d-7015-401f-900b-d065938f4507-kube-api-access-m7ghr\") pod \"a5c9f74d-7015-401f-900b-d065938f4507\" (UID: \"a5c9f74d-7015-401f-900b-d065938f4507\") "
Nov 02 02:08:42.649903 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:42.649613    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c9f74d-7015-401f-900b-d065938f4507-openshift-service-ca" pod "a5c9f74d-7015-401f-900b-d065938f4507" (UID: "a5c9f74d-7015-401f-900b-d065938f4507"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:42.663423 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:42.661638    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c9f74d-7015-401f-900b-d065938f4507-kube-api-access-m7ghr" pod "a5c9f74d-7015-401f-900b-d065938f4507" (UID: "a5c9f74d-7015-401f-900b-d065938f4507"). InnerVolumeSpecName "kube-api-access-m7ghr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:42.663856 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:42.663811    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c9f74d-7015-401f-900b-d065938f4507-maas-api-tls" pod "a5c9f74d-7015-401f-900b-d065938f4507" (UID: "a5c9f74d-7015-401f-900b-d065938f4507"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:08:42.664733 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:42.664687    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c9f74d-7015-401f-900b-d065938f4507-metrics-certs" pod "a5c9f74d-7015-401f-900b-d065938f4507" (UID: "a5c9f74d-7015-401f-900b-d065938f4507"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:08:42.734489 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:42.734459    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7ghr\" (UniqueName: \"kubernetes.io/projected/a5c9f74d-7015-401f-900b-d065938f4507-kube-api-access-m7ghr\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:42.734489 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:42.734488    5875 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5c9f74d-7015-401f-900b-d065938f4507-metrics-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:42.734675 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:42.734498    5875 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5c9f74d-7015-401f-900b-d065938f4507-openshift-service-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:42.734675 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:42.734506    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5c9f74d-7015-401f-900b-d065938f4507-maas-api-tls\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:42.996584 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:42.996549    5875 generic.go:356] "Generic (PLEG): container finished" podID="a5c9f74d-7015-401f-900b-d065938f4507" containerID="b896f0500cc649a6f494f98756c4e1f88726f2b8f64db991e95ea5998bde6093" exitCode=0
Nov 02 02:08:42.996740 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:42.996597    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd"
Nov 02 02:08:42.996740 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:42.996628    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd" event={"ID":"a5c9f74d-7015-401f-900b-d065938f4507","Type":"ContainerDied","Data":"b896f0500cc649a6f494f98756c4e1f88726f2b8f64db991e95ea5998bde6093"}
Nov 02 02:08:42.996740 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:42.996683    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd" event={"ID":"a5c9f74d-7015-401f-900b-d065938f4507","Type":"ContainerDied","Data":"a3c1c35555491070a5def47d62614d42b1592885ad51ddb942588f8442b48318"}
Nov 02 02:08:42.996740 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:42.996706    5875 scope.go:122] "RemoveContainer" containerID="b896f0500cc649a6f494f98756c4e1f88726f2b8f64db991e95ea5998bde6093"
Nov 02 02:08:43.006395 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:43.006375    5875 scope.go:122] "RemoveContainer" containerID="b896f0500cc649a6f494f98756c4e1f88726f2b8f64db991e95ea5998bde6093"
Nov 02 02:08:43.006676 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:08:43.006656    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b896f0500cc649a6f494f98756c4e1f88726f2b8f64db991e95ea5998bde6093\": container with ID starting with b896f0500cc649a6f494f98756c4e1f88726f2b8f64db991e95ea5998bde6093 not found: ID does not exist" containerID="b896f0500cc649a6f494f98756c4e1f88726f2b8f64db991e95ea5998bde6093"
Nov 02 02:08:43.006745 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:43.006686    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b896f0500cc649a6f494f98756c4e1f88726f2b8f64db991e95ea5998bde6093"} err="failed to get container status \"b896f0500cc649a6f494f98756c4e1f88726f2b8f64db991e95ea5998bde6093\": rpc error: code = NotFound desc = could not find container \"b896f0500cc649a6f494f98756c4e1f88726f2b8f64db991e95ea5998bde6093\": container with ID starting with b896f0500cc649a6f494f98756c4e1f88726f2b8f64db991e95ea5998bde6093 not found: ID does not exist"
Nov 02 02:08:43.218728 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:43.218688    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd"]
Nov 02 02:08:43.228144 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:43.228080    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8e79d7-765d668d6-wndvd"]
Nov 02 02:08:44.694255 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:44.694208    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5c9f74d-7015-401f-900b-d065938f4507" path="/var/lib/kubelet/pods/a5c9f74d-7015-401f-900b-d065938f4507/volumes"
Nov 02 02:08:49.887136 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:49.887102    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm"]
Nov 02 02:08:49.887584 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:49.887382    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm" podUID="2572e86e-fe1c-4b20-ba3f-9eb3270b6127" containerName="maas-api" containerID="cri-o://bfd1599b2d4968997bc3e992a4ce24407aacd87e7ab882192b8644954b818096" gracePeriod=30
Nov 02 02:08:50.043855 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:50.043824    5875 generic.go:356] "Generic (PLEG): container finished" podID="2572e86e-fe1c-4b20-ba3f-9eb3270b6127" containerID="bfd1599b2d4968997bc3e992a4ce24407aacd87e7ab882192b8644954b818096" exitCode=0
Nov 02 02:08:50.044021 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:50.043908    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm" event={"ID":"2572e86e-fe1c-4b20-ba3f-9eb3270b6127","Type":"ContainerDied","Data":"bfd1599b2d4968997bc3e992a4ce24407aacd87e7ab882192b8644954b818096"}
Nov 02 02:08:50.178700 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:50.178677    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm"
Nov 02 02:08:50.244929 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:50.244900    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-openshift-service-ca\") pod \"2572e86e-fe1c-4b20-ba3f-9eb3270b6127\" (UID: \"2572e86e-fe1c-4b20-ba3f-9eb3270b6127\") "
Nov 02 02:08:50.245084 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:50.244967    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-metrics-certs\") pod \"2572e86e-fe1c-4b20-ba3f-9eb3270b6127\" (UID: \"2572e86e-fe1c-4b20-ba3f-9eb3270b6127\") "
Nov 02 02:08:50.245084 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:50.244997    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-kube-api-access-k9kx6\" (UniqueName: \"kubernetes.io/projected/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-kube-api-access-k9kx6\") pod \"2572e86e-fe1c-4b20-ba3f-9eb3270b6127\" (UID: \"2572e86e-fe1c-4b20-ba3f-9eb3270b6127\") "
Nov 02 02:08:50.245084 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:50.245022    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-maas-api-tls\") pod \"2572e86e-fe1c-4b20-ba3f-9eb3270b6127\" (UID: \"2572e86e-fe1c-4b20-ba3f-9eb3270b6127\") "
Nov 02 02:08:50.245364 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:50.245342    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-openshift-service-ca" pod "2572e86e-fe1c-4b20-ba3f-9eb3270b6127" (UID: "2572e86e-fe1c-4b20-ba3f-9eb3270b6127"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:50.245676 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:50.245653    5875 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-openshift-service-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:50.247460 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:50.247435    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-metrics-certs" pod "2572e86e-fe1c-4b20-ba3f-9eb3270b6127" (UID: "2572e86e-fe1c-4b20-ba3f-9eb3270b6127"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:08:50.247549 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:50.247435    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-kube-api-access-k9kx6" pod "2572e86e-fe1c-4b20-ba3f-9eb3270b6127" (UID: "2572e86e-fe1c-4b20-ba3f-9eb3270b6127"). InnerVolumeSpecName "kube-api-access-k9kx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:50.247549 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:50.247503    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-maas-api-tls" pod "2572e86e-fe1c-4b20-ba3f-9eb3270b6127" (UID: "2572e86e-fe1c-4b20-ba3f-9eb3270b6127"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:08:50.356124 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:50.356090    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-maas-api-tls\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:50.356124 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:50.356124    5875 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-metrics-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:50.356124 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:50.356139    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9kx6\" (UniqueName: \"kubernetes.io/projected/2572e86e-fe1c-4b20-ba3f-9eb3270b6127-kube-api-access-k9kx6\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:51.050619 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.050539    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm"
Nov 02 02:08:51.051097 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.050536    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm" event={"ID":"2572e86e-fe1c-4b20-ba3f-9eb3270b6127","Type":"ContainerDied","Data":"bd26469a0b187c9b0c4ba899f689560a4ab2d24c19dae329f71c5e19e7941af9"}
Nov 02 02:08:51.051097 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.050671    5875 scope.go:122] "RemoveContainer" containerID="bfd1599b2d4968997bc3e992a4ce24407aacd87e7ab882192b8644954b818096"
Nov 02 02:08:51.214925 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.214887    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm"]
Nov 02 02:08:51.264275 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.264241    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d1405a-b74f98554-9nzxm"]
Nov 02 02:08:51.590260 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.588195    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm"]
Nov 02 02:08:51.591275 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.591245    5875 cpu_manager.go:414] "containerMap: removing container" podUID="43d6edfb-c755-4afd-9237-1d4ce70352f1" containerName="payload-pre-processing"
Nov 02 02:08:51.591275 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.591276    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d6edfb-c755-4afd-9237-1d4ce70352f1" containerName="payload-pre-processing"
Nov 02 02:08:51.591486 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.591298    5875 cpu_manager.go:414] "containerMap: removing container" podUID="a5c9f74d-7015-401f-900b-d065938f4507" containerName="maas-api"
Nov 02 02:08:51.591486 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.591307    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c9f74d-7015-401f-900b-d065938f4507" containerName="maas-api"
Nov 02 02:08:51.591486 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.591337    5875 cpu_manager.go:414] "containerMap: removing container" podUID="409169ea-5827-4946-9836-638b0bd62614" containerName="payload-processing"
Nov 02 02:08:51.591486 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.591345    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="409169ea-5827-4946-9836-638b0bd62614" containerName="payload-processing"
Nov 02 02:08:51.591486 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.591387    5875 cpu_manager.go:414] "containerMap: removing container" podUID="2572e86e-fe1c-4b20-ba3f-9eb3270b6127" containerName="maas-api"
Nov 02 02:08:51.591486 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.591395    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2572e86e-fe1c-4b20-ba3f-9eb3270b6127" containerName="maas-api"
Nov 02 02:08:51.591844 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.591577    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="409169ea-5827-4946-9836-638b0bd62614" containerName="payload-processing"
Nov 02 02:08:51.591844 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.591600    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="2572e86e-fe1c-4b20-ba3f-9eb3270b6127" containerName="maas-api"
Nov 02 02:08:51.591844 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.591617    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d6edfb-c755-4afd-9237-1d4ce70352f1" containerName="payload-pre-processing"
Nov 02 02:08:51.591844 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.591628    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5c9f74d-7015-401f-900b-d065938f4507" containerName="maas-api"
Nov 02 02:08:51.598537 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.598458    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm"
Nov 02 02:08:51.637436 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.637403    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"]
Nov 02 02:08:51.637655 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.637633    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7" podUID="9493769f-3e6b-44f1-aa30-03081e4db485" containerName="istio-proxy" containerID="cri-o://e880d4607cd78c7eb96e9c76a71ade86737977b35139bf0db1c1a9a7a07d8eef" gracePeriod=30
Nov 02 02:08:51.650545 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.650515    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5700d3d1-serving-cert\""
Nov 02 02:08:51.654022 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.654000    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm"]
Nov 02 02:08:51.677576 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.677534    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af93eb7b-4ac4-4619-a3b9-958c82ab2280-openshift-service-ca\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm\" (UID: \"af93eb7b-4ac4-4619-a3b9-958c82ab2280\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm"
Nov 02 02:08:51.677758 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.677675    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv2fs\" (UniqueName: \"kubernetes.io/projected/af93eb7b-4ac4-4619-a3b9-958c82ab2280-kube-api-access-mv2fs\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm\" (UID: \"af93eb7b-4ac4-4619-a3b9-958c82ab2280\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm"
Nov 02 02:08:51.677858 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.677831    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af93eb7b-4ac4-4619-a3b9-958c82ab2280-maas-api-tls\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm\" (UID: \"af93eb7b-4ac4-4619-a3b9-958c82ab2280\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm"
Nov 02 02:08:51.677904 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.677888    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af93eb7b-4ac4-4619-a3b9-958c82ab2280-metrics-certs\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm\" (UID: \"af93eb7b-4ac4-4619-a3b9-958c82ab2280\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm"
Nov 02 02:08:51.779680 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.779644    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv2fs\" (UniqueName: \"kubernetes.io/projected/af93eb7b-4ac4-4619-a3b9-958c82ab2280-kube-api-access-mv2fs\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm\" (UID: \"af93eb7b-4ac4-4619-a3b9-958c82ab2280\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm"
Nov 02 02:08:51.779866 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.779805    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af93eb7b-4ac4-4619-a3b9-958c82ab2280-maas-api-tls\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm\" (UID: \"af93eb7b-4ac4-4619-a3b9-958c82ab2280\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm"
Nov 02 02:08:51.779866 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.779850    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af93eb7b-4ac4-4619-a3b9-958c82ab2280-metrics-certs\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm\" (UID: \"af93eb7b-4ac4-4619-a3b9-958c82ab2280\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm"
Nov 02 02:08:51.780021 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.779927    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af93eb7b-4ac4-4619-a3b9-958c82ab2280-openshift-service-ca\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm\" (UID: \"af93eb7b-4ac4-4619-a3b9-958c82ab2280\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm"
Nov 02 02:08:51.780021 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:08:51.779971    5875 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-serving-cert: secret "maas-api-e2e-mt-5700d3d1-serving-cert" not found
Nov 02 02:08:51.780126 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:08:51.780032    5875 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af93eb7b-4ac4-4619-a3b9-958c82ab2280-maas-api-tls podName:af93eb7b-4ac4-4619-a3b9-958c82ab2280 nodeName:}" failed. No retries permitted until 2026-11-02 02:08:52.280015652 +0000 UTC m=+2179.721052131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/af93eb7b-4ac4-4619-a3b9-958c82ab2280-maas-api-tls") pod "maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm" (UID: "af93eb7b-4ac4-4619-a3b9-958c82ab2280") : secret "maas-api-e2e-mt-5700d3d1-serving-cert" not found
Nov 02 02:08:51.780854 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.780823    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af93eb7b-4ac4-4619-a3b9-958c82ab2280-openshift-service-ca\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm\" (UID: \"af93eb7b-4ac4-4619-a3b9-958c82ab2280\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm"
Nov 02 02:08:51.783057 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.783028    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af93eb7b-4ac4-4619-a3b9-958c82ab2280-metrics-certs\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm\" (UID: \"af93eb7b-4ac4-4619-a3b9-958c82ab2280\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm"
Nov 02 02:08:51.897870 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:51.897783    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv2fs\" (UniqueName: \"kubernetes.io/projected/af93eb7b-4ac4-4619-a3b9-958c82ab2280-kube-api-access-mv2fs\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm\" (UID: \"af93eb7b-4ac4-4619-a3b9-958c82ab2280\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm"
Nov 02 02:08:52.301694 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:52.301662    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af93eb7b-4ac4-4619-a3b9-958c82ab2280-maas-api-tls\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm\" (UID: \"af93eb7b-4ac4-4619-a3b9-958c82ab2280\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm"
Nov 02 02:08:52.304476 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:52.304453    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af93eb7b-4ac4-4619-a3b9-958c82ab2280-maas-api-tls\") pod \"maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm\" (UID: \"af93eb7b-4ac4-4619-a3b9-958c82ab2280\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm"
Nov 02 02:08:52.519197 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:52.519169    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm"
Nov 02 02:08:52.695303 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:52.695268    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2572e86e-fe1c-4b20-ba3f-9eb3270b6127" path="/var/lib/kubelet/pods/2572e86e-fe1c-4b20-ba3f-9eb3270b6127/volumes"
Nov 02 02:08:52.853009 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:52.852928    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"]
Nov 02 02:08:52.853251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:52.853197    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb" podUID="0931244f-232a-4588-8b26-eb3b5b5773c2" containerName="istio-proxy" containerID="cri-o://dd6bc2e03f40e3c3027cbe573c08b7b5134eef05070a973b40aed85d1eaab534" gracePeriod=30
Nov 02 02:08:52.895819 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:52.895777    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm"]
Nov 02 02:08:52.934409 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:08:52.934375    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf93eb7b_4ac4_4619_a3b9_958c82ab2280.slice/crio-0d61918aef155d31140ba10d0e7ea934ac3098a6ce49b3cc1864d14d48f0a7b9 WatchSource:0}: Error finding container 0d61918aef155d31140ba10d0e7ea934ac3098a6ce49b3cc1864d14d48f0a7b9: Status 404 returned error can't find the container with id 0d61918aef155d31140ba10d0e7ea934ac3098a6ce49b3cc1864d14d48f0a7b9
Nov 02 02:08:53.064559 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:53.064529    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm" event={"ID":"af93eb7b-4ac4-4619-a3b9-958c82ab2280","Type":"ContainerStarted","Data":"0d61918aef155d31140ba10d0e7ea934ac3098a6ce49b3cc1864d14d48f0a7b9"}
Nov 02 02:08:54.072358 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:54.072270    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm" event={"ID":"af93eb7b-4ac4-4619-a3b9-958c82ab2280","Type":"ContainerStarted","Data":"9ba97d1d50dc4de164f7efc430b90b87ee8bb372b6e75145787adcf3f43b831d"}
Nov 02 02:08:54.072747 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:54.072450    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm"
Nov 02 02:08:54.191530 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:54.191474    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm" podStartSLOduration=2.343834704 podStartE2EDuration="3.191461168s" totalImagesPullingTime="847.626464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:08:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:08:54.189041181 +0000 UTC m=+2181.630077679" watchObservedRunningTime="2026-11-02 02:08:54.191461168 +0000 UTC m=+2181.632497854"
Nov 02 02:08:56.531208 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:56.531164    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw"]
Nov 02 02:08:56.531688 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:56.531497    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw" podUID="2d772318-f18c-4308-9e87-182a635baff0" containerName="maas-api" containerID="cri-o://5ea86199c509316a94721b75bec2085a4fa77ba73b36e17a4c5d6394af7c15f2" gracePeriod=30
Nov 02 02:08:56.854957 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:56.854902    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw"
Nov 02 02:08:56.969975 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:56.969949    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d772318-f18c-4308-9e87-182a635baff0-kube-api-access-tgrjz\" (UniqueName: \"kubernetes.io/projected/2d772318-f18c-4308-9e87-182a635baff0-kube-api-access-tgrjz\") pod \"2d772318-f18c-4308-9e87-182a635baff0\" (UID: \"2d772318-f18c-4308-9e87-182a635baff0\") "
Nov 02 02:08:56.970323 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:56.970271    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d772318-f18c-4308-9e87-182a635baff0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d772318-f18c-4308-9e87-182a635baff0-openshift-service-ca\") pod \"2d772318-f18c-4308-9e87-182a635baff0\" (UID: \"2d772318-f18c-4308-9e87-182a635baff0\") "
Nov 02 02:08:56.970587 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:56.970557    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d772318-f18c-4308-9e87-182a635baff0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d772318-f18c-4308-9e87-182a635baff0-maas-api-tls\") pod \"2d772318-f18c-4308-9e87-182a635baff0\" (UID: \"2d772318-f18c-4308-9e87-182a635baff0\") "
Nov 02 02:08:56.970703 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:56.970684    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d772318-f18c-4308-9e87-182a635baff0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d772318-f18c-4308-9e87-182a635baff0-metrics-certs\") pod \"2d772318-f18c-4308-9e87-182a635baff0\" (UID: \"2d772318-f18c-4308-9e87-182a635baff0\") "
Nov 02 02:08:56.970778 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:56.970735    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d772318-f18c-4308-9e87-182a635baff0-openshift-service-ca" pod "2d772318-f18c-4308-9e87-182a635baff0" (UID: "2d772318-f18c-4308-9e87-182a635baff0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:56.971409 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:56.971377    5875 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d772318-f18c-4308-9e87-182a635baff0-openshift-service-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:56.973676 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:56.973644    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d772318-f18c-4308-9e87-182a635baff0-maas-api-tls" pod "2d772318-f18c-4308-9e87-182a635baff0" (UID: "2d772318-f18c-4308-9e87-182a635baff0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:08:56.974057 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:56.974033    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d772318-f18c-4308-9e87-182a635baff0-metrics-certs" pod "2d772318-f18c-4308-9e87-182a635baff0" (UID: "2d772318-f18c-4308-9e87-182a635baff0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:08:56.974138 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:56.974060    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:08:56.976380 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:56.976346    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d772318-f18c-4308-9e87-182a635baff0-kube-api-access-tgrjz" pod "2d772318-f18c-4308-9e87-182a635baff0" (UID: "2d772318-f18c-4308-9e87-182a635baff0"). InnerVolumeSpecName "kube-api-access-tgrjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:57.071917 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.071838    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-credential-socket\") pod \"9493769f-3e6b-44f1-aa30-03081e4db485\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") "
Nov 02 02:08:57.071917 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.071895    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-istio-envoy\") pod \"9493769f-3e6b-44f1-aa30-03081e4db485\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") "
Nov 02 02:08:57.071917 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.071919    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9493769f-3e6b-44f1-aa30-03081e4db485-kube-api-access-8h7pr\" (UniqueName: \"kubernetes.io/projected/9493769f-3e6b-44f1-aa30-03081e4db485-kube-api-access-8h7pr\") pod \"9493769f-3e6b-44f1-aa30-03081e4db485\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") "
Nov 02 02:08:57.072207 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.071972    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9493769f-3e6b-44f1-aa30-03081e4db485-istio-token\" (UniqueName: \"kubernetes.io/projected/9493769f-3e6b-44f1-aa30-03081e4db485-istio-token\") pod \"9493769f-3e6b-44f1-aa30-03081e4db485\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") "
Nov 02 02:08:57.072207 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.071988    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-istio-data\") pod \"9493769f-3e6b-44f1-aa30-03081e4db485\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") "
Nov 02 02:08:57.072207 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.072006    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-workload-certs\") pod \"9493769f-3e6b-44f1-aa30-03081e4db485\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") "
Nov 02 02:08:57.072207 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.072124    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9493769f-3e6b-44f1-aa30-03081e4db485-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9493769f-3e6b-44f1-aa30-03081e4db485-istiod-ca-cert\") pod \"9493769f-3e6b-44f1-aa30-03081e4db485\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") "
Nov 02 02:08:57.072207 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.072158    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9493769f-3e6b-44f1-aa30-03081e4db485-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9493769f-3e6b-44f1-aa30-03081e4db485-istio-podinfo\") pod \"9493769f-3e6b-44f1-aa30-03081e4db485\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") "
Nov 02 02:08:57.072529 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.072210    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-workload-socket\") pod \"9493769f-3e6b-44f1-aa30-03081e4db485\" (UID: \"9493769f-3e6b-44f1-aa30-03081e4db485\") "
Nov 02 02:08:57.072529 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.072211    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-credential-socket" pod "9493769f-3e6b-44f1-aa30-03081e4db485" (UID: "9493769f-3e6b-44f1-aa30-03081e4db485"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:57.072529 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.072322    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-istio-data" pod "9493769f-3e6b-44f1-aa30-03081e4db485" (UID: "9493769f-3e6b-44f1-aa30-03081e4db485"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:57.072529 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.072431    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-workload-certs" pod "9493769f-3e6b-44f1-aa30-03081e4db485" (UID: "9493769f-3e6b-44f1-aa30-03081e4db485"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:57.072746 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.072576    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-workload-socket" pod "9493769f-3e6b-44f1-aa30-03081e4db485" (UID: "9493769f-3e6b-44f1-aa30-03081e4db485"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:57.072746 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.072607    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9493769f-3e6b-44f1-aa30-03081e4db485-istiod-ca-cert" pod "9493769f-3e6b-44f1-aa30-03081e4db485" (UID: "9493769f-3e6b-44f1-aa30-03081e4db485"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:57.073025 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.072854    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:57.073025 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.072870    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:57.073025 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.072881    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9493769f-3e6b-44f1-aa30-03081e4db485-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:57.073025 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.072890    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:57.073025 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.072899    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d772318-f18c-4308-9e87-182a635baff0-maas-api-tls\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:57.073025 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.072906    5875 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d772318-f18c-4308-9e87-182a635baff0-metrics-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:57.073025 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.072915    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:57.073025 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.072924    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgrjz\" (UniqueName: \"kubernetes.io/projected/2d772318-f18c-4308-9e87-182a635baff0-kube-api-access-tgrjz\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:57.074274 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.074209    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-istio-envoy" pod "9493769f-3e6b-44f1-aa30-03081e4db485" (UID: "9493769f-3e6b-44f1-aa30-03081e4db485"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:57.074415 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.074395    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9493769f-3e6b-44f1-aa30-03081e4db485-istio-podinfo" pod "9493769f-3e6b-44f1-aa30-03081e4db485" (UID: "9493769f-3e6b-44f1-aa30-03081e4db485"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:08:57.074534 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.074512    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9493769f-3e6b-44f1-aa30-03081e4db485-istio-token" pod "9493769f-3e6b-44f1-aa30-03081e4db485" (UID: "9493769f-3e6b-44f1-aa30-03081e4db485"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:57.074645 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.074630    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9493769f-3e6b-44f1-aa30-03081e4db485-kube-api-access-8h7pr" pod "9493769f-3e6b-44f1-aa30-03081e4db485" (UID: "9493769f-3e6b-44f1-aa30-03081e4db485"). InnerVolumeSpecName "kube-api-access-8h7pr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:57.115123 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.115090    5875 generic.go:356] "Generic (PLEG): container finished" podID="2d772318-f18c-4308-9e87-182a635baff0" containerID="5ea86199c509316a94721b75bec2085a4fa77ba73b36e17a4c5d6394af7c15f2" exitCode=0
Nov 02 02:08:57.115123 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.115159    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw"
Nov 02 02:08:57.115123 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.115361    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw" event={"ID":"2d772318-f18c-4308-9e87-182a635baff0","Type":"ContainerDied","Data":"5ea86199c509316a94721b75bec2085a4fa77ba73b36e17a4c5d6394af7c15f2"}
Nov 02 02:08:57.115123 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.115422    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw" event={"ID":"2d772318-f18c-4308-9e87-182a635baff0","Type":"ContainerDied","Data":"9b2e9a0749fdd943541b0df77ff9ce30808a361e370b49947dec6ab2ad46fc24"}
Nov 02 02:08:57.115123 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.115458    5875 scope.go:122] "RemoveContainer" containerID="5ea86199c509316a94721b75bec2085a4fa77ba73b36e17a4c5d6394af7c15f2"
Nov 02 02:08:57.119988 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.119963    5875 generic.go:356] "Generic (PLEG): container finished" podID="9493769f-3e6b-44f1-aa30-03081e4db485" containerID="e880d4607cd78c7eb96e9c76a71ade86737977b35139bf0db1c1a9a7a07d8eef" exitCode=0
Nov 02 02:08:57.120127 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.120040    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"
Nov 02 02:08:57.120127 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.120048    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7" event={"ID":"9493769f-3e6b-44f1-aa30-03081e4db485","Type":"ContainerDied","Data":"e880d4607cd78c7eb96e9c76a71ade86737977b35139bf0db1c1a9a7a07d8eef"}
Nov 02 02:08:57.120127 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.120089    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7" event={"ID":"9493769f-3e6b-44f1-aa30-03081e4db485","Type":"ContainerDied","Data":"8cd688c8c8a26594574a8f3727416d9b753f41d3d85f0f099357e60f173f50f1"}
Nov 02 02:08:57.129399 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.129379    5875 scope.go:122] "RemoveContainer" containerID="5ea86199c509316a94721b75bec2085a4fa77ba73b36e17a4c5d6394af7c15f2"
Nov 02 02:08:57.129893 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:08:57.129786    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ea86199c509316a94721b75bec2085a4fa77ba73b36e17a4c5d6394af7c15f2\": container with ID starting with 5ea86199c509316a94721b75bec2085a4fa77ba73b36e17a4c5d6394af7c15f2 not found: ID does not exist" containerID="5ea86199c509316a94721b75bec2085a4fa77ba73b36e17a4c5d6394af7c15f2"
Nov 02 02:08:57.129893 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.129826    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ea86199c509316a94721b75bec2085a4fa77ba73b36e17a4c5d6394af7c15f2"} err="failed to get container status \"5ea86199c509316a94721b75bec2085a4fa77ba73b36e17a4c5d6394af7c15f2\": rpc error: code = NotFound desc = could not find container \"5ea86199c509316a94721b75bec2085a4fa77ba73b36e17a4c5d6394af7c15f2\": container with ID starting with 5ea86199c509316a94721b75bec2085a4fa77ba73b36e17a4c5d6394af7c15f2 not found: ID does not exist"
Nov 02 02:08:57.129893 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.129851    5875 scope.go:122] "RemoveContainer" containerID="e880d4607cd78c7eb96e9c76a71ade86737977b35139bf0db1c1a9a7a07d8eef"
Nov 02 02:08:57.140743 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.140722    5875 scope.go:122] "RemoveContainer" containerID="581f5dfbafadadcab3095717730bee01dfc9eed93b78c8791fed28e4dc574558"
Nov 02 02:08:57.151690 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.151670    5875 scope.go:122] "RemoveContainer" containerID="e880d4607cd78c7eb96e9c76a71ade86737977b35139bf0db1c1a9a7a07d8eef"
Nov 02 02:08:57.152032 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:08:57.152005    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e880d4607cd78c7eb96e9c76a71ade86737977b35139bf0db1c1a9a7a07d8eef\": container with ID starting with e880d4607cd78c7eb96e9c76a71ade86737977b35139bf0db1c1a9a7a07d8eef not found: ID does not exist" containerID="e880d4607cd78c7eb96e9c76a71ade86737977b35139bf0db1c1a9a7a07d8eef"
Nov 02 02:08:57.152173 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.152148    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e880d4607cd78c7eb96e9c76a71ade86737977b35139bf0db1c1a9a7a07d8eef"} err="failed to get container status \"e880d4607cd78c7eb96e9c76a71ade86737977b35139bf0db1c1a9a7a07d8eef\": rpc error: code = NotFound desc = could not find container \"e880d4607cd78c7eb96e9c76a71ade86737977b35139bf0db1c1a9a7a07d8eef\": container with ID starting with e880d4607cd78c7eb96e9c76a71ade86737977b35139bf0db1c1a9a7a07d8eef not found: ID does not exist"
Nov 02 02:08:57.152339 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.152323    5875 scope.go:122] "RemoveContainer" containerID="581f5dfbafadadcab3095717730bee01dfc9eed93b78c8791fed28e4dc574558"
Nov 02 02:08:57.152800 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:08:57.152736    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"581f5dfbafadadcab3095717730bee01dfc9eed93b78c8791fed28e4dc574558\": container with ID starting with 581f5dfbafadadcab3095717730bee01dfc9eed93b78c8791fed28e4dc574558 not found: ID does not exist" containerID="581f5dfbafadadcab3095717730bee01dfc9eed93b78c8791fed28e4dc574558"
Nov 02 02:08:57.152800 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.152768    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"581f5dfbafadadcab3095717730bee01dfc9eed93b78c8791fed28e4dc574558"} err="failed to get container status \"581f5dfbafadadcab3095717730bee01dfc9eed93b78c8791fed28e4dc574558\": rpc error: code = NotFound desc = could not find container \"581f5dfbafadadcab3095717730bee01dfc9eed93b78c8791fed28e4dc574558\": container with ID starting with 581f5dfbafadadcab3095717730bee01dfc9eed93b78c8791fed28e4dc574558 not found: ID does not exist"
Nov 02 02:08:57.174517 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.174494    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9493769f-3e6b-44f1-aa30-03081e4db485-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:57.174612 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.174520    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h7pr\" (UniqueName: \"kubernetes.io/projected/9493769f-3e6b-44f1-aa30-03081e4db485-kube-api-access-8h7pr\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:57.174612 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.174532    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9493769f-3e6b-44f1-aa30-03081e4db485-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:57.174612 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.174541    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9493769f-3e6b-44f1-aa30-03081e4db485-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:57.249399 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.249368    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"]
Nov 02 02:08:57.254171 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.254147    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-8e79d7-openshift-default-77f48cdc87-prcs7"]
Nov 02 02:08:57.361830 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.361749    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw"]
Nov 02 02:08:57.366189 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:57.366160    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11f26c-9d55b9f59-rvdcw"]
Nov 02 02:08:58.129869 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.129800    5875 generic.go:356] "Generic (PLEG): container finished" podID="0931244f-232a-4588-8b26-eb3b5b5773c2" containerID="dd6bc2e03f40e3c3027cbe573c08b7b5134eef05070a973b40aed85d1eaab534" exitCode=0
Nov 02 02:08:58.130302 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.129872    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb" event={"ID":"0931244f-232a-4588-8b26-eb3b5b5773c2","Type":"ContainerDied","Data":"dd6bc2e03f40e3c3027cbe573c08b7b5134eef05070a973b40aed85d1eaab534"}
Nov 02 02:08:58.130302 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.129922    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb" event={"ID":"0931244f-232a-4588-8b26-eb3b5b5773c2","Type":"ContainerDied","Data":"d81cbc22e97e5c0cb298b8609ff77e9d112f385aebd60d1ccf9d27036d93ec25"}
Nov 02 02:08:58.130302 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.129949    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d81cbc22e97e5c0cb298b8609ff77e9d112f385aebd60d1ccf9d27036d93ec25"
Nov 02 02:08:58.130302 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.129963    5875 scope.go:122] "RemoveContainer" containerID="929b255fdf84f63a74a2f0290af1e9bb6a189e5bb6512e7a1665d741a414e661"
Nov 02 02:08:58.132008 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.131987    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:08:58.289920 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.289887    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-podinfo\") pod \"0931244f-232a-4588-8b26-eb3b5b5773c2\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") "
Nov 02 02:08:58.290120 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.289942    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-workload-certs\") pod \"0931244f-232a-4588-8b26-eb3b5b5773c2\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") "
Nov 02 02:08:58.290120 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.290005    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0931244f-232a-4588-8b26-eb3b5b5773c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0931244f-232a-4588-8b26-eb3b5b5773c2-istiod-ca-cert\") pod \"0931244f-232a-4588-8b26-eb3b5b5773c2\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") "
Nov 02 02:08:58.290120 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.290071    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-credential-socket\") pod \"0931244f-232a-4588-8b26-eb3b5b5773c2\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") "
Nov 02 02:08:58.290311 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.290129    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-token\" (UniqueName: \"kubernetes.io/projected/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-token\") pod \"0931244f-232a-4588-8b26-eb3b5b5773c2\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") "
Nov 02 02:08:58.290311 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.290170    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-envoy\") pod \"0931244f-232a-4588-8b26-eb3b5b5773c2\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") "
Nov 02 02:08:58.290311 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.290210    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-workload-socket\") pod \"0931244f-232a-4588-8b26-eb3b5b5773c2\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") "
Nov 02 02:08:58.290311 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.290307    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0931244f-232a-4588-8b26-eb3b5b5773c2-kube-api-access-svrm8\" (UniqueName: \"kubernetes.io/projected/0931244f-232a-4588-8b26-eb3b5b5773c2-kube-api-access-svrm8\") pod \"0931244f-232a-4588-8b26-eb3b5b5773c2\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") "
Nov 02 02:08:58.290528 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.290349    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-workload-certs" pod "0931244f-232a-4588-8b26-eb3b5b5773c2" (UID: "0931244f-232a-4588-8b26-eb3b5b5773c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:58.290528 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.290364    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-data\") pod \"0931244f-232a-4588-8b26-eb3b5b5773c2\" (UID: \"0931244f-232a-4588-8b26-eb3b5b5773c2\") "
Nov 02 02:08:58.290633 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.290582    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-workload-socket" pod "0931244f-232a-4588-8b26-eb3b5b5773c2" (UID: "0931244f-232a-4588-8b26-eb3b5b5773c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:58.290684 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.290620    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0931244f-232a-4588-8b26-eb3b5b5773c2-istiod-ca-cert" pod "0931244f-232a-4588-8b26-eb3b5b5773c2" (UID: "0931244f-232a-4588-8b26-eb3b5b5773c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:58.293261 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.290835    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-credential-socket" pod "0931244f-232a-4588-8b26-eb3b5b5773c2" (UID: "0931244f-232a-4588-8b26-eb3b5b5773c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:58.293261 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.291175    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:58.293261 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.291199    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0931244f-232a-4588-8b26-eb3b5b5773c2-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:58.293261 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.291215    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:58.293261 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.291247    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:58.293261 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.291385    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-data" pod "0931244f-232a-4588-8b26-eb3b5b5773c2" (UID: "0931244f-232a-4588-8b26-eb3b5b5773c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:58.293261 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.292853    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-podinfo" pod "0931244f-232a-4588-8b26-eb3b5b5773c2" (UID: "0931244f-232a-4588-8b26-eb3b5b5773c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:08:58.293261 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.293183    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-token" pod "0931244f-232a-4588-8b26-eb3b5b5773c2" (UID: "0931244f-232a-4588-8b26-eb3b5b5773c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:58.293703 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.293680    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0931244f-232a-4588-8b26-eb3b5b5773c2-kube-api-access-svrm8" pod "0931244f-232a-4588-8b26-eb3b5b5773c2" (UID: "0931244f-232a-4588-8b26-eb3b5b5773c2"). InnerVolumeSpecName "kube-api-access-svrm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:58.293782 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.293767    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-envoy" pod "0931244f-232a-4588-8b26-eb3b5b5773c2" (UID: "0931244f-232a-4588-8b26-eb3b5b5773c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:58.392559 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.392475    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:58.392559 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.392504    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:58.392559 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.392513    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:58.392559 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.392523    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svrm8\" (UniqueName: \"kubernetes.io/projected/0931244f-232a-4588-8b26-eb3b5b5773c2-kube-api-access-svrm8\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:58.392559 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.392533    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0931244f-232a-4588-8b26-eb3b5b5773c2-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:58.698899 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.698800    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d772318-f18c-4308-9e87-182a635baff0" path="/var/lib/kubelet/pods/2d772318-f18c-4308-9e87-182a635baff0/volumes"
Nov 02 02:08:58.699347 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:58.699330    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9493769f-3e6b-44f1-aa30-03081e4db485" path="/var/lib/kubelet/pods/9493769f-3e6b-44f1-aa30-03081e4db485/volumes"
Nov 02 02:08:59.138916 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:59.138886    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"
Nov 02 02:08:59.391277 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:59.390276    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"]
Nov 02 02:08:59.400285 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:59.393126    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"]
Nov 02 02:08:59.400285 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:59.393405    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9" podUID="f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" containerName="istio-proxy" containerID="cri-o://b81ed9d044bfa0f851dae0ea861de42672d2af0932b168b1fe84fb38e843d954" gracePeriod=30
Nov 02 02:08:59.422996 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:08:59.422953    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-47eb54-openshift-default-5b547bdd79-wqprb"]
Nov 02 02:09:00.084386 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:00.084352    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm"
Nov 02 02:09:00.694628 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:00.694598    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0931244f-232a-4588-8b26-eb3b5b5773c2" path="/var/lib/kubelet/pods/0931244f-232a-4588-8b26-eb3b5b5773c2/volumes"
Nov 02 02:09:01.096292 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.096259    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"]
Nov 02 02:09:01.096796 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.096782    5875 cpu_manager.go:414] "containerMap: removing container" podUID="0931244f-232a-4588-8b26-eb3b5b5773c2" containerName="istio-proxy"
Nov 02 02:09:01.096842 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.096797    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0931244f-232a-4588-8b26-eb3b5b5773c2" containerName="istio-proxy"
Nov 02 02:09:01.096842 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.096813    5875 cpu_manager.go:414] "containerMap: removing container" podUID="9493769f-3e6b-44f1-aa30-03081e4db485" containerName="istio-proxy"
Nov 02 02:09:01.096842 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.096818    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9493769f-3e6b-44f1-aa30-03081e4db485" containerName="istio-proxy"
Nov 02 02:09:01.096842 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.096830    5875 cpu_manager.go:414] "containerMap: removing container" podUID="2d772318-f18c-4308-9e87-182a635baff0" containerName="maas-api"
Nov 02 02:09:01.096842 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.096835    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d772318-f18c-4308-9e87-182a635baff0" containerName="maas-api"
Nov 02 02:09:01.097001 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.096858    5875 cpu_manager.go:414] "containerMap: removing container" podUID="9493769f-3e6b-44f1-aa30-03081e4db485" containerName="istio-proxy"
Nov 02 02:09:01.097001 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.096863    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9493769f-3e6b-44f1-aa30-03081e4db485" containerName="istio-proxy"
Nov 02 02:09:01.097001 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.096952    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="0931244f-232a-4588-8b26-eb3b5b5773c2" containerName="istio-proxy"
Nov 02 02:09:01.097001 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.096960    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="0931244f-232a-4588-8b26-eb3b5b5773c2" containerName="istio-proxy"
Nov 02 02:09:01.097001 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.096970    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="9493769f-3e6b-44f1-aa30-03081e4db485" containerName="istio-proxy"
Nov 02 02:09:01.097001 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.096976    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d772318-f18c-4308-9e87-182a635baff0" containerName="maas-api"
Nov 02 02:09:01.101919 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.101898    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.170465 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.170419    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-092079-openshift-default-dockercfg-wjkzk\""
Nov 02 02:09:01.207753 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.207722    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"]
Nov 02 02:09:01.262246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.259113    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-data\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.262246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.259322    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-credential-socket\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.262246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.259362    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-workload-socket\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.262246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.259399    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6dl9\" (UniqueName: \"kubernetes.io/projected/d12fc012-f64e-4504-85dc-0bbabf28cb68-kube-api-access-s6dl9\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.262246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.259458    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d12fc012-f64e-4504-85dc-0bbabf28cb68-istiod-ca-cert\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.262246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.259491    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-podinfo\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.262246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.259535    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-workload-certs\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.262246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.259559    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-token\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.262246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.259667    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-envoy\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.360859 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.360778    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-podinfo\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.360859 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.360822    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-workload-certs\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.360859 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.360839    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-token\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.361142 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.360875    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-envoy\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.361142 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.360911    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-data\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.361307 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.361144    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-credential-socket\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.361307 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.361196    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-workload-socket\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.361307 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.361259    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-workload-certs\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.361307 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.361275    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6dl9\" (UniqueName: \"kubernetes.io/projected/d12fc012-f64e-4504-85dc-0bbabf28cb68-kube-api-access-s6dl9\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.361584 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.361351    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d12fc012-f64e-4504-85dc-0bbabf28cb68-istiod-ca-cert\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.361584 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.361377    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-data\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.361584 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.361524    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-credential-socket\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.361927 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.361594    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-workload-socket\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.361998 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.361952    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d12fc012-f64e-4504-85dc-0bbabf28cb68-istiod-ca-cert\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.363309 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.363290    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-envoy\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.363622 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.363604    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-podinfo\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.423599 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.423568    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-token\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.423820 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.423796    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6dl9\" (UniqueName: \"kubernetes.io/projected/d12fc012-f64e-4504-85dc-0bbabf28cb68-kube-api-access-s6dl9\") pod \"e2e-api-a-092079-openshift-default-59469cb5ff-npkzt\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") " pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:01.715403 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:01.715379    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:02.149819 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:02.149741    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"]
Nov 02 02:09:02.200643 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:09:02.200607    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12fc012_f64e_4504_85dc_0bbabf28cb68.slice/crio-422fd9608f19a14527800d19ebeb871b74aa2b0de3c8a57034e3c0377160620d WatchSource:0}: Error finding container 422fd9608f19a14527800d19ebeb871b74aa2b0de3c8a57034e3c0377160620d: Status 404 returned error can't find the container with id 422fd9608f19a14527800d19ebeb871b74aa2b0de3c8a57034e3c0377160620d
Nov 02 02:09:02.203068 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:02.203031    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:09:02.203201 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:02.203116    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:09:02.203201 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:02.203161    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:09:03.189256 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:03.189212    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt" event={"ID":"d12fc012-f64e-4504-85dc-0bbabf28cb68","Type":"ContainerStarted","Data":"9404db203db3eb9bf9266ed3eb94df6ab4e6a820631fcedd4d864dffeaeb4ff4"}
Nov 02 02:09:03.189636 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:03.189275    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt" event={"ID":"d12fc012-f64e-4504-85dc-0bbabf28cb68","Type":"ContainerStarted","Data":"422fd9608f19a14527800d19ebeb871b74aa2b0de3c8a57034e3c0377160620d"}
Nov 02 02:09:03.360957 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:03.360904    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt" podStartSLOduration=2.360886998 podStartE2EDuration="2.360886998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:09:03.35888328 +0000 UTC m=+2190.799919776" watchObservedRunningTime="2026-11-02 02:09:03.360886998 +0000 UTC m=+2190.801923496"
Nov 02 02:09:03.716233 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:03.716192    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:03.721779 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:03.721746    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:04.196054 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.195979    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:04.196881 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.196863    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:09:04.667429 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.667399    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:09:04.710343 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.710313    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-podinfo\") pod \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") "
Nov 02 02:09:04.710508 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.710384    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-data\") pod \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") "
Nov 02 02:09:04.710508 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.710424    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-envoy\") pod \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") "
Nov 02 02:09:04.710508 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.710481    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-kube-api-access-4ldj5\" (UniqueName: \"kubernetes.io/projected/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-kube-api-access-4ldj5\") pod \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") "
Nov 02 02:09:04.710697 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.710538    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istiod-ca-cert\") pod \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") "
Nov 02 02:09:04.710697 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.710569    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-token\" (UniqueName: \"kubernetes.io/projected/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-token\") pod \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") "
Nov 02 02:09:04.710697 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.710624    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-workload-socket\") pod \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") "
Nov 02 02:09:04.710697 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.710680    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-data" pod "f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" (UID: "f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:04.710912 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.710702    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-workload-certs\") pod \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") "
Nov 02 02:09:04.710912 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.710787    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-credential-socket\") pod \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\" (UID: \"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e\") "
Nov 02 02:09:04.711006 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.710934    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-workload-socket" pod "f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" (UID: "f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:04.711006 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.710998    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-workload-certs" pod "f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" (UID: "f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:04.711127 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.711031    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istiod-ca-cert" pod "f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" (UID: "f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:09:04.711202 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.711185    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-credential-socket" pod "f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" (UID: "f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:04.711549 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.711526    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:04.711608 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.711557    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:04.711608 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.711575    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:04.711608 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.711590    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:04.711608 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.711604    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:04.712797 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.712768    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-envoy" pod "f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" (UID: "f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:04.712956 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.712941    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-kube-api-access-4ldj5" pod "f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" (UID: "f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e"). InnerVolumeSpecName "kube-api-access-4ldj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:04.713276 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.713253    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-token" pod "f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" (UID: "f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:04.713379 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.713270    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-podinfo" pod "f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" (UID: "f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:09:04.813603 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.813527    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:04.813603 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.813567    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ldj5\" (UniqueName: \"kubernetes.io/projected/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-kube-api-access-4ldj5\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:04.813603 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.813584    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:04.813603 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:04.813598    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:05.203941 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:05.203841    5875 generic.go:356] "Generic (PLEG): container finished" podID="f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" containerID="b81ed9d044bfa0f851dae0ea861de42672d2af0932b168b1fe84fb38e843d954" exitCode=0
Nov 02 02:09:05.203941 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:05.203903    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"
Nov 02 02:09:05.203941 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:05.203917    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9" event={"ID":"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e","Type":"ContainerDied","Data":"b81ed9d044bfa0f851dae0ea861de42672d2af0932b168b1fe84fb38e843d954"}
Nov 02 02:09:05.204533 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:05.203978    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9" event={"ID":"f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e","Type":"ContainerDied","Data":"e5ae2d9265129749e7d00c445699bf2df12c2a0a26516ee9048bcd317e1f9167"}
Nov 02 02:09:05.204533 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:05.204000    5875 scope.go:122] "RemoveContainer" containerID="b81ed9d044bfa0f851dae0ea861de42672d2af0932b168b1fe84fb38e843d954"
Nov 02 02:09:05.215565 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:05.215550    5875 scope.go:122] "RemoveContainer" containerID="ab0e4330b8ab6fc2f3d8cd2eb1bdf8d62ef21d2d23bd3bcc23dec820a8eddbab"
Nov 02 02:09:05.224568 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:05.224548    5875 scope.go:122] "RemoveContainer" containerID="b81ed9d044bfa0f851dae0ea861de42672d2af0932b168b1fe84fb38e843d954"
Nov 02 02:09:05.224823 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:09:05.224803    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b81ed9d044bfa0f851dae0ea861de42672d2af0932b168b1fe84fb38e843d954\": container with ID starting with b81ed9d044bfa0f851dae0ea861de42672d2af0932b168b1fe84fb38e843d954 not found: ID does not exist" containerID="b81ed9d044bfa0f851dae0ea861de42672d2af0932b168b1fe84fb38e843d954"
Nov 02 02:09:05.224877 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:05.224830    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b81ed9d044bfa0f851dae0ea861de42672d2af0932b168b1fe84fb38e843d954"} err="failed to get container status \"b81ed9d044bfa0f851dae0ea861de42672d2af0932b168b1fe84fb38e843d954\": rpc error: code = NotFound desc = could not find container \"b81ed9d044bfa0f851dae0ea861de42672d2af0932b168b1fe84fb38e843d954\": container with ID starting with b81ed9d044bfa0f851dae0ea861de42672d2af0932b168b1fe84fb38e843d954 not found: ID does not exist"
Nov 02 02:09:05.224877 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:05.224849    5875 scope.go:122] "RemoveContainer" containerID="ab0e4330b8ab6fc2f3d8cd2eb1bdf8d62ef21d2d23bd3bcc23dec820a8eddbab"
Nov 02 02:09:05.225080 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:09:05.225065    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab0e4330b8ab6fc2f3d8cd2eb1bdf8d62ef21d2d23bd3bcc23dec820a8eddbab\": container with ID starting with ab0e4330b8ab6fc2f3d8cd2eb1bdf8d62ef21d2d23bd3bcc23dec820a8eddbab not found: ID does not exist" containerID="ab0e4330b8ab6fc2f3d8cd2eb1bdf8d62ef21d2d23bd3bcc23dec820a8eddbab"
Nov 02 02:09:05.225123 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:05.225085    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0e4330b8ab6fc2f3d8cd2eb1bdf8d62ef21d2d23bd3bcc23dec820a8eddbab"} err="failed to get container status \"ab0e4330b8ab6fc2f3d8cd2eb1bdf8d62ef21d2d23bd3bcc23dec820a8eddbab\": rpc error: code = NotFound desc = could not find container \"ab0e4330b8ab6fc2f3d8cd2eb1bdf8d62ef21d2d23bd3bcc23dec820a8eddbab\": container with ID starting with ab0e4330b8ab6fc2f3d8cd2eb1bdf8d62ef21d2d23bd3bcc23dec820a8eddbab not found: ID does not exist"
Nov 02 02:09:05.370867 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:05.370826    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"]
Nov 02 02:09:05.421320 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:05.421282    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d1405a-openshift-default-88998b695-d82k9"]
Nov 02 02:09:06.163262 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.163191    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"]
Nov 02 02:09:06.163532 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.163503    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92" podUID="4650bde4-c321-40aa-bc8e-af7bc0dd3311" containerName="istio-proxy" containerID="cri-o://b446b7bc1e91f16ddbf4398719abaf05c97cec9a6a4b04f69eeebd8b0bd4ee82" gracePeriod=30
Nov 02 02:09:06.525613 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.525576    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"]
Nov 02 02:09:06.526375 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.526353    5875 cpu_manager.go:414] "containerMap: removing container" podUID="f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" containerName="istio-proxy"
Nov 02 02:09:06.526375 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.526375    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" containerName="istio-proxy"
Nov 02 02:09:06.526534 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.526463    5875 cpu_manager.go:414] "containerMap: removing container" podUID="f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" containerName="istio-proxy"
Nov 02 02:09:06.526534 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.526472    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" containerName="istio-proxy"
Nov 02 02:09:06.526534 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.526493    5875 cpu_manager.go:414] "containerMap: removing container" podUID="0931244f-232a-4588-8b26-eb3b5b5773c2" containerName="istio-proxy"
Nov 02 02:09:06.526534 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.526502    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0931244f-232a-4588-8b26-eb3b5b5773c2" containerName="istio-proxy"
Nov 02 02:09:06.526749 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.526637    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" containerName="istio-proxy"
Nov 02 02:09:06.526749 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.526655    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="9493769f-3e6b-44f1-aa30-03081e4db485" containerName="istio-proxy"
Nov 02 02:09:06.531491 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.531470    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.591888 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.591853    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"]
Nov 02 02:09:06.628091 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.628058    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4211c0-openshift-default-dockercfg-c7dtw\""
Nov 02 02:09:06.637551 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.637523    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-credential-socket\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.637669 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.637570    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh42p\" (UniqueName: \"kubernetes.io/projected/93d9b083-f014-4998-91b1-a90d3cce473c-kube-api-access-zh42p\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.637711 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.637678    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93d9b083-f014-4998-91b1-a90d3cce473c-istiod-ca-cert\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.637711 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.637704    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-workload-certs\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.637818 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.637781    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-istio-data\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.637877 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.637817    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93d9b083-f014-4998-91b1-a90d3cce473c-istio-podinfo\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.637957 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.637936    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-istio-envoy\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.638018 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.637977    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93d9b083-f014-4998-91b1-a90d3cce473c-istio-token\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.638060 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.638030    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-workload-socket\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.695387 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.695355    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" path="/var/lib/kubelet/pods/f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e/volumes"
Nov 02 02:09:06.739570 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.739534    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-credential-socket\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.739859 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.739831    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh42p\" (UniqueName: \"kubernetes.io/projected/93d9b083-f014-4998-91b1-a90d3cce473c-kube-api-access-zh42p\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.739981 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.739920    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93d9b083-f014-4998-91b1-a90d3cce473c-istiod-ca-cert\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.739981 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.739951    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-workload-certs\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.740085 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.739991    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-istio-data\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.740085 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.740032    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93d9b083-f014-4998-91b1-a90d3cce473c-istio-podinfo\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.740198 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.740111    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-istio-envoy\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.740198 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.740144    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93d9b083-f014-4998-91b1-a90d3cce473c-istio-token\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.740198 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.740148    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-credential-socket\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.740198 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.740188    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-workload-socket\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.740431 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.740391    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-istio-data\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.740663 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.740635    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-workload-socket\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.742730 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.742359    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-workload-certs\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.743033 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.743008    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93d9b083-f014-4998-91b1-a90d3cce473c-istiod-ca-cert\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.747083 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.745575    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-istio-envoy\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.747083 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.746632    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93d9b083-f014-4998-91b1-a90d3cce473c-istio-podinfo\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.804375 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.804316    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh42p\" (UniqueName: \"kubernetes.io/projected/93d9b083-f014-4998-91b1-a90d3cce473c-kube-api-access-zh42p\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.804881 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.804866    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93d9b083-f014-4998-91b1-a90d3cce473c-istio-token\") pod \"e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") " pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:06.844626 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:06.844602    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:07.286855 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:07.286139    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"]
Nov 02 02:09:07.292008 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:07.291971    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:09:07.292116 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:07.292040    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:09:07.292116 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:07.292070    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:09:08.227869 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:08.227829    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn" event={"ID":"93d9b083-f014-4998-91b1-a90d3cce473c","Type":"ContainerStarted","Data":"28ed2b891f119c8f2542e0c5598765f9ec0988beec097bc0617f6ffceb5c4f5c"}
Nov 02 02:09:08.228251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:08.227914    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn" event={"ID":"93d9b083-f014-4998-91b1-a90d3cce473c","Type":"ContainerStarted","Data":"4dc49dc6f5906862e676b5b65356113ad6f1f8f23a9906ea4df1f9089001e48e"}
Nov 02 02:09:08.343967 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:08.343936    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfc6dbfe-wcfp8"]
Nov 02 02:09:08.344628 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:08.344613    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ed0d40-2b1a-4e8c-b8c7-abff4f0d972e" containerName="istio-proxy"
Nov 02 02:09:08.348069 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:08.348046    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc6dbfe-wcfp8"
Nov 02 02:09:08.363835 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:08.363811    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba1ad30-5e76-4452-821d-ca8309c88af6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfc6dbfe-wcfp8\" (UID: \"4ba1ad30-5e76-4452-821d-ca8309c88af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc6dbfe-wcfp8"
Nov 02 02:09:08.363991 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:08.363873    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dn4f\" (UniqueName: \"kubernetes.io/projected/4ba1ad30-5e76-4452-821d-ca8309c88af6-kube-api-access-8dn4f\") pod \"maas-api-revoke-sub-bfc6dbfe-wcfp8\" (UID: \"4ba1ad30-5e76-4452-821d-ca8309c88af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc6dbfe-wcfp8"
Nov 02 02:09:08.364038 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:08.363978    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba1ad30-5e76-4452-821d-ca8309c88af6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfc6dbfe-wcfp8\" (UID: \"4ba1ad30-5e76-4452-821d-ca8309c88af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc6dbfe-wcfp8"
Nov 02 02:09:08.404037 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:08.403989    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:09:08.404201 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:08.404167    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfc6dbfe-wcfp8"]
Nov 02 02:09:08.451162 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:08.451107    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn" podStartSLOduration=2.451091913 podStartE2EDuration="2.451091913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:09:08.403992944 +0000 UTC m=+2195.845029442" watchObservedRunningTime="2026-11-02 02:09:08.451091913 +0000 UTC m=+2195.892128407"
Nov 02 02:09:08.465248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:08.465189    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba1ad30-5e76-4452-821d-ca8309c88af6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfc6dbfe-wcfp8\" (UID: \"4ba1ad30-5e76-4452-821d-ca8309c88af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc6dbfe-wcfp8"
Nov 02 02:09:08.465412 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:08.465321    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dn4f\" (UniqueName: \"kubernetes.io/projected/4ba1ad30-5e76-4452-821d-ca8309c88af6-kube-api-access-8dn4f\") pod \"maas-api-revoke-sub-bfc6dbfe-wcfp8\" (UID: \"4ba1ad30-5e76-4452-821d-ca8309c88af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc6dbfe-wcfp8"
Nov 02 02:09:08.465412 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:08.465352    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba1ad30-5e76-4452-821d-ca8309c88af6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfc6dbfe-wcfp8\" (UID: \"4ba1ad30-5e76-4452-821d-ca8309c88af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc6dbfe-wcfp8"
Nov 02 02:09:08.466001 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:08.465972    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba1ad30-5e76-4452-821d-ca8309c88af6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfc6dbfe-wcfp8\" (UID: \"4ba1ad30-5e76-4452-821d-ca8309c88af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc6dbfe-wcfp8"
Nov 02 02:09:08.467708 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:08.467687    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba1ad30-5e76-4452-821d-ca8309c88af6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfc6dbfe-wcfp8\" (UID: \"4ba1ad30-5e76-4452-821d-ca8309c88af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc6dbfe-wcfp8"
Nov 02 02:09:08.503291 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:08.503242    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:09:08.546061 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:08.546017    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dn4f\" (UniqueName: \"kubernetes.io/projected/4ba1ad30-5e76-4452-821d-ca8309c88af6-kube-api-access-8dn4f\") pod \"maas-api-revoke-sub-bfc6dbfe-wcfp8\" (UID: \"4ba1ad30-5e76-4452-821d-ca8309c88af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc6dbfe-wcfp8"
Nov 02 02:09:08.663768 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:08.663741    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc6dbfe-wcfp8"
Nov 02 02:09:08.845427 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:08.845353    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:08.850690 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:08.850669    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:09.084555 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:09.084355    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfc6dbfe-wcfp8"]
Nov 02 02:09:09.087703 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:09:09.087670    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba1ad30_5e76_4452_821d_ca8309c88af6.slice/crio-19c97378554cdfa5c466e2d4721589c17e3585fdd4bb7b5f277f7255a32fc26e WatchSource:0}: Error finding container 19c97378554cdfa5c466e2d4721589c17e3585fdd4bb7b5f277f7255a32fc26e: Status 404 returned error can't find the container with id 19c97378554cdfa5c466e2d4721589c17e3585fdd4bb7b5f277f7255a32fc26e
Nov 02 02:09:09.250907 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:09.250855    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc6dbfe-wcfp8" event={"ID":"4ba1ad30-5e76-4452-821d-ca8309c88af6","Type":"ContainerStarted","Data":"19c97378554cdfa5c466e2d4721589c17e3585fdd4bb7b5f277f7255a32fc26e"}
Nov 02 02:09:09.251395 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:09.251165    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:09.252318 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:09.252300    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:09:10.258284 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:10.258170    5875 generic.go:356] "Generic (PLEG): container finished" podID="4ba1ad30-5e76-4452-821d-ca8309c88af6" containerID="b7d24dd00ab46d819cb67eafd5bb141c4d248c282f134ce6588acfe7dcf3c039" exitCode=0
Nov 02 02:09:10.259396 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:10.259372    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc6dbfe-wcfp8" event={"ID":"4ba1ad30-5e76-4452-821d-ca8309c88af6","Type":"ContainerDied","Data":"b7d24dd00ab46d819cb67eafd5bb141c4d248c282f134ce6588acfe7dcf3c039"}
Nov 02 02:09:11.326982 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:09:11.326101    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:09:11.440070 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.440008    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc6dbfe-wcfp8"
Nov 02 02:09:11.475856 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.475826    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:09:11.508053 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.508027    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ba1ad30-5e76-4452-821d-ca8309c88af6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba1ad30-5e76-4452-821d-ca8309c88af6-maas-api-xks-ca\") pod \"4ba1ad30-5e76-4452-821d-ca8309c88af6\" (UID: \"4ba1ad30-5e76-4452-821d-ca8309c88af6\") "
Nov 02 02:09:11.508202 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.508076    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istiod-ca-cert\") pod \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") "
Nov 02 02:09:11.508202 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.508106    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba1ad30-5e76-4452-821d-ca8309c88af6-kube-api-access-8dn4f\" (UniqueName: \"kubernetes.io/projected/4ba1ad30-5e76-4452-821d-ca8309c88af6-kube-api-access-8dn4f\") pod \"4ba1ad30-5e76-4452-821d-ca8309c88af6\" (UID: \"4ba1ad30-5e76-4452-821d-ca8309c88af6\") "
Nov 02 02:09:11.508202 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.508128    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4650bde4-c321-40aa-bc8e-af7bc0dd3311-kube-api-access-lvmrw\" (UniqueName: \"kubernetes.io/projected/4650bde4-c321-40aa-bc8e-af7bc0dd3311-kube-api-access-lvmrw\") pod \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") "
Nov 02 02:09:11.508202 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.508165    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ba1ad30-5e76-4452-821d-ca8309c88af6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba1ad30-5e76-4452-821d-ca8309c88af6-maas-api-openshift-ca\") pod \"4ba1ad30-5e76-4452-821d-ca8309c88af6\" (UID: \"4ba1ad30-5e76-4452-821d-ca8309c88af6\") "
Nov 02 02:09:11.508202 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.508190    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-data\") pod \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") "
Nov 02 02:09:11.508518 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.508302    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-workload-certs\") pod \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") "
Nov 02 02:09:11.508518 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.508377    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-podinfo\") pod \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") "
Nov 02 02:09:11.508518 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.508451    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-envoy\") pod \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") "
Nov 02 02:09:11.508518 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.508484    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-token\" (UniqueName: \"kubernetes.io/projected/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-token\") pod \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") "
Nov 02 02:09:11.508806 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.508583    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-credential-socket\") pod \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") "
Nov 02 02:09:11.508806 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.508630    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-workload-socket\") pod \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\" (UID: \"4650bde4-c321-40aa-bc8e-af7bc0dd3311\") "
Nov 02 02:09:11.508806 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.508479    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istiod-ca-cert" pod "4650bde4-c321-40aa-bc8e-af7bc0dd3311" (UID: "4650bde4-c321-40aa-bc8e-af7bc0dd3311"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:09:11.508806 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.508592    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-data" pod "4650bde4-c321-40aa-bc8e-af7bc0dd3311" (UID: "4650bde4-c321-40aa-bc8e-af7bc0dd3311"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:11.508806 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.508652    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-workload-certs" pod "4650bde4-c321-40aa-bc8e-af7bc0dd3311" (UID: "4650bde4-c321-40aa-bc8e-af7bc0dd3311"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:11.508806 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.508772    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba1ad30-5e76-4452-821d-ca8309c88af6-maas-api-openshift-ca" pod "4ba1ad30-5e76-4452-821d-ca8309c88af6" (UID: "4ba1ad30-5e76-4452-821d-ca8309c88af6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:09:11.509310 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.508995    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-credential-socket" pod "4650bde4-c321-40aa-bc8e-af7bc0dd3311" (UID: "4650bde4-c321-40aa-bc8e-af7bc0dd3311"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:11.509310 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.509093    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-workload-socket" pod "4650bde4-c321-40aa-bc8e-af7bc0dd3311" (UID: "4650bde4-c321-40aa-bc8e-af7bc0dd3311"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:11.509498 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.509475    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba1ad30-5e76-4452-821d-ca8309c88af6-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:11.509568 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.509505    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:11.509568 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.509521    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:11.509568 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.509534    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:11.509568 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.509549    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:11.509568 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.509563    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:11.510846 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.510817    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-podinfo" pod "4650bde4-c321-40aa-bc8e-af7bc0dd3311" (UID: "4650bde4-c321-40aa-bc8e-af7bc0dd3311"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:09:11.510846 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.510825    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4650bde4-c321-40aa-bc8e-af7bc0dd3311-kube-api-access-lvmrw" pod "4650bde4-c321-40aa-bc8e-af7bc0dd3311" (UID: "4650bde4-c321-40aa-bc8e-af7bc0dd3311"). InnerVolumeSpecName "kube-api-access-lvmrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:11.511294 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.511280    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-token" pod "4650bde4-c321-40aa-bc8e-af7bc0dd3311" (UID: "4650bde4-c321-40aa-bc8e-af7bc0dd3311"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:11.511454 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.511434    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba1ad30-5e76-4452-821d-ca8309c88af6-kube-api-access-8dn4f" pod "4ba1ad30-5e76-4452-821d-ca8309c88af6" (UID: "4ba1ad30-5e76-4452-821d-ca8309c88af6"). InnerVolumeSpecName "kube-api-access-8dn4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:11.511525 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.511494    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ba1ad30-5e76-4452-821d-ca8309c88af6-maas-api-xks-ca" pod "4ba1ad30-5e76-4452-821d-ca8309c88af6" (UID: "4ba1ad30-5e76-4452-821d-ca8309c88af6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:09:11.511525 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.511508    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-envoy" pod "4650bde4-c321-40aa-bc8e-af7bc0dd3311" (UID: "4650bde4-c321-40aa-bc8e-af7bc0dd3311"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:11.610851 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.610818    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dn4f\" (UniqueName: \"kubernetes.io/projected/4ba1ad30-5e76-4452-821d-ca8309c88af6-kube-api-access-8dn4f\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:11.610851 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.610844    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvmrw\" (UniqueName: \"kubernetes.io/projected/4650bde4-c321-40aa-bc8e-af7bc0dd3311-kube-api-access-lvmrw\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:11.610851 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.610858    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:11.611093 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.610868    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:11.611093 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.610889    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4650bde4-c321-40aa-bc8e-af7bc0dd3311-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:11.611093 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.610904    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba1ad30-5e76-4452-821d-ca8309c88af6-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:11.710971 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:11.710929    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"]
Nov 02 02:09:12.273596 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:12.273557    5875 generic.go:356] "Generic (PLEG): container finished" podID="4650bde4-c321-40aa-bc8e-af7bc0dd3311" containerID="b446b7bc1e91f16ddbf4398719abaf05c97cec9a6a4b04f69eeebd8b0bd4ee82" exitCode=0
Nov 02 02:09:12.273769 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:12.273624    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"
Nov 02 02:09:12.273769 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:12.273642    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92" event={"ID":"4650bde4-c321-40aa-bc8e-af7bc0dd3311","Type":"ContainerDied","Data":"b446b7bc1e91f16ddbf4398719abaf05c97cec9a6a4b04f69eeebd8b0bd4ee82"}
Nov 02 02:09:12.273769 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:12.273731    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92" event={"ID":"4650bde4-c321-40aa-bc8e-af7bc0dd3311","Type":"ContainerDied","Data":"d0705bb93989569a26653bb3773a6efef9d1441ef3a942f833ef0f852a2243fa"}
Nov 02 02:09:12.273769 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:12.273759    5875 scope.go:122] "RemoveContainer" containerID="b446b7bc1e91f16ddbf4398719abaf05c97cec9a6a4b04f69eeebd8b0bd4ee82"
Nov 02 02:09:12.275539 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:12.275280    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc6dbfe-wcfp8" event={"ID":"4ba1ad30-5e76-4452-821d-ca8309c88af6","Type":"ContainerDied","Data":"19c97378554cdfa5c466e2d4721589c17e3585fdd4bb7b5f277f7255a32fc26e"}
Nov 02 02:09:12.275539 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:12.275327    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19c97378554cdfa5c466e2d4721589c17e3585fdd4bb7b5f277f7255a32fc26e"
Nov 02 02:09:12.275539 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:12.275395    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc6dbfe-wcfp8"
Nov 02 02:09:12.285882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:12.285863    5875 scope.go:122] "RemoveContainer" containerID="d3e2a3f5397eefe994e1bbd3e203ce3edb73f56cff8cf6f0757f72524a2ffc58"
Nov 02 02:09:12.295844 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:12.295823    5875 scope.go:122] "RemoveContainer" containerID="b446b7bc1e91f16ddbf4398719abaf05c97cec9a6a4b04f69eeebd8b0bd4ee82"
Nov 02 02:09:12.296141 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:09:12.296119    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b446b7bc1e91f16ddbf4398719abaf05c97cec9a6a4b04f69eeebd8b0bd4ee82\": container with ID starting with b446b7bc1e91f16ddbf4398719abaf05c97cec9a6a4b04f69eeebd8b0bd4ee82 not found: ID does not exist" containerID="b446b7bc1e91f16ddbf4398719abaf05c97cec9a6a4b04f69eeebd8b0bd4ee82"
Nov 02 02:09:12.296201 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:12.296154    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b446b7bc1e91f16ddbf4398719abaf05c97cec9a6a4b04f69eeebd8b0bd4ee82"} err="failed to get container status \"b446b7bc1e91f16ddbf4398719abaf05c97cec9a6a4b04f69eeebd8b0bd4ee82\": rpc error: code = NotFound desc = could not find container \"b446b7bc1e91f16ddbf4398719abaf05c97cec9a6a4b04f69eeebd8b0bd4ee82\": container with ID starting with b446b7bc1e91f16ddbf4398719abaf05c97cec9a6a4b04f69eeebd8b0bd4ee82 not found: ID does not exist"
Nov 02 02:09:12.296201 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:12.296177    5875 scope.go:122] "RemoveContainer" containerID="d3e2a3f5397eefe994e1bbd3e203ce3edb73f56cff8cf6f0757f72524a2ffc58"
Nov 02 02:09:12.296559 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:09:12.296538    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3e2a3f5397eefe994e1bbd3e203ce3edb73f56cff8cf6f0757f72524a2ffc58\": container with ID starting with d3e2a3f5397eefe994e1bbd3e203ce3edb73f56cff8cf6f0757f72524a2ffc58 not found: ID does not exist" containerID="d3e2a3f5397eefe994e1bbd3e203ce3edb73f56cff8cf6f0757f72524a2ffc58"
Nov 02 02:09:12.296608 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:12.296566    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e2a3f5397eefe994e1bbd3e203ce3edb73f56cff8cf6f0757f72524a2ffc58"} err="failed to get container status \"d3e2a3f5397eefe994e1bbd3e203ce3edb73f56cff8cf6f0757f72524a2ffc58\": rpc error: code = NotFound desc = could not find container \"d3e2a3f5397eefe994e1bbd3e203ce3edb73f56cff8cf6f0757f72524a2ffc58\": container with ID starting with d3e2a3f5397eefe994e1bbd3e203ce3edb73f56cff8cf6f0757f72524a2ffc58 not found: ID does not exist"
Nov 02 02:09:12.491385 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:12.491337    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"]
Nov 02 02:09:12.539584 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:12.539548    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-11f26c-openshift-default-85bffdc4bf-vmk92"]
Nov 02 02:09:12.696320 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:12.696290    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4650bde4-c321-40aa-bc8e-af7bc0dd3311" path="/var/lib/kubelet/pods/4650bde4-c321-40aa-bc8e-af7bc0dd3311/volumes"
Nov 02 02:09:13.360008 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.359967    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s"]
Nov 02 02:09:13.360614 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.360596    5875 cpu_manager.go:414] "containerMap: removing container" podUID="4650bde4-c321-40aa-bc8e-af7bc0dd3311" containerName="istio-proxy"
Nov 02 02:09:13.360733 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.360616    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4650bde4-c321-40aa-bc8e-af7bc0dd3311" containerName="istio-proxy"
Nov 02 02:09:13.360733 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.360661    5875 cpu_manager.go:414] "containerMap: removing container" podUID="4ba1ad30-5e76-4452-821d-ca8309c88af6" containerName="revoke-keys"
Nov 02 02:09:13.360733 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.360669    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba1ad30-5e76-4452-821d-ca8309c88af6" containerName="revoke-keys"
Nov 02 02:09:13.360897 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.360836    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="4650bde4-c321-40aa-bc8e-af7bc0dd3311" containerName="istio-proxy"
Nov 02 02:09:13.360897 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.360856    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba1ad30-5e76-4452-821d-ca8309c88af6" containerName="revoke-keys"
Nov 02 02:09:13.365783 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.365759    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s"
Nov 02 02:09:13.419985 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.419950    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-092079\""
Nov 02 02:09:13.448670 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.448633    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/051907ab-4e37-4298-9291-0df5fa65d735-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s\" (UID: \"051907ab-4e37-4298-9291-0df5fa65d735\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s"
Nov 02 02:09:13.449101 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.449081    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6pvv\" (UniqueName: \"kubernetes.io/projected/051907ab-4e37-4298-9291-0df5fa65d735-kube-api-access-g6pvv\") pod \"payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s\" (UID: \"051907ab-4e37-4298-9291-0df5fa65d735\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s"
Nov 02 02:09:13.476955 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.476901    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s"]
Nov 02 02:09:13.477201 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.477104    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-092079-dockercfg-2vcjg\""
Nov 02 02:09:13.482343 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.482313    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-092079-9c6765d77-fxqv2"]
Nov 02 02:09:13.483269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.483249    5875 cpu_manager.go:414] "containerMap: removing container" podUID="4650bde4-c321-40aa-bc8e-af7bc0dd3311" containerName="istio-proxy"
Nov 02 02:09:13.483335 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.483273    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4650bde4-c321-40aa-bc8e-af7bc0dd3311" containerName="istio-proxy"
Nov 02 02:09:13.488096 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.488067    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="4650bde4-c321-40aa-bc8e-af7bc0dd3311" containerName="istio-proxy"
Nov 02 02:09:13.493244 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.492149    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-092079-9c6765d77-fxqv2"
Nov 02 02:09:13.543376 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.543346    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-092079-9c6765d77-fxqv2"]
Nov 02 02:09:13.551348 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.551314    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6pvv\" (UniqueName: \"kubernetes.io/projected/051907ab-4e37-4298-9291-0df5fa65d735-kube-api-access-g6pvv\") pod \"payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s\" (UID: \"051907ab-4e37-4298-9291-0df5fa65d735\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s"
Nov 02 02:09:13.551710 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.551689    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ae28f8-e723-4bb3-b368-6fb9881d24bf-plugins-config-volume\") pod \"payload-processing-e2e-api-a-092079-9c6765d77-fxqv2\" (UID: \"c5ae28f8-e723-4bb3-b368-6fb9881d24bf\") " pod="openshift-ingress/payload-processing-e2e-api-a-092079-9c6765d77-fxqv2"
Nov 02 02:09:13.551810 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.551760    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfx9l\" (UniqueName: \"kubernetes.io/projected/c5ae28f8-e723-4bb3-b368-6fb9881d24bf-kube-api-access-bfx9l\") pod \"payload-processing-e2e-api-a-092079-9c6765d77-fxqv2\" (UID: \"c5ae28f8-e723-4bb3-b368-6fb9881d24bf\") " pod="openshift-ingress/payload-processing-e2e-api-a-092079-9c6765d77-fxqv2"
Nov 02 02:09:13.551922 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.551899    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/051907ab-4e37-4298-9291-0df5fa65d735-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s\" (UID: \"051907ab-4e37-4298-9291-0df5fa65d735\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s"
Nov 02 02:09:13.552890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.552865    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/051907ab-4e37-4298-9291-0df5fa65d735-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s\" (UID: \"051907ab-4e37-4298-9291-0df5fa65d735\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s"
Nov 02 02:09:13.613181 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.613100    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6pvv\" (UniqueName: \"kubernetes.io/projected/051907ab-4e37-4298-9291-0df5fa65d735-kube-api-access-g6pvv\") pod \"payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s\" (UID: \"051907ab-4e37-4298-9291-0df5fa65d735\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s"
Nov 02 02:09:13.653744 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.653712    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ae28f8-e723-4bb3-b368-6fb9881d24bf-plugins-config-volume\") pod \"payload-processing-e2e-api-a-092079-9c6765d77-fxqv2\" (UID: \"c5ae28f8-e723-4bb3-b368-6fb9881d24bf\") " pod="openshift-ingress/payload-processing-e2e-api-a-092079-9c6765d77-fxqv2"
Nov 02 02:09:13.653903 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.653774    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfx9l\" (UniqueName: \"kubernetes.io/projected/c5ae28f8-e723-4bb3-b368-6fb9881d24bf-kube-api-access-bfx9l\") pod \"payload-processing-e2e-api-a-092079-9c6765d77-fxqv2\" (UID: \"c5ae28f8-e723-4bb3-b368-6fb9881d24bf\") " pod="openshift-ingress/payload-processing-e2e-api-a-092079-9c6765d77-fxqv2"
Nov 02 02:09:13.654397 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.654376    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ae28f8-e723-4bb3-b368-6fb9881d24bf-plugins-config-volume\") pod \"payload-processing-e2e-api-a-092079-9c6765d77-fxqv2\" (UID: \"c5ae28f8-e723-4bb3-b368-6fb9881d24bf\") " pod="openshift-ingress/payload-processing-e2e-api-a-092079-9c6765d77-fxqv2"
Nov 02 02:09:13.679603 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.679568    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s"
Nov 02 02:09:13.714682 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.714644    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfx9l\" (UniqueName: \"kubernetes.io/projected/c5ae28f8-e723-4bb3-b368-6fb9881d24bf-kube-api-access-bfx9l\") pod \"payload-processing-e2e-api-a-092079-9c6765d77-fxqv2\" (UID: \"c5ae28f8-e723-4bb3-b368-6fb9881d24bf\") " pod="openshift-ingress/payload-processing-e2e-api-a-092079-9c6765d77-fxqv2"
Nov 02 02:09:13.808917 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:13.808886    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-092079-9c6765d77-fxqv2"
Nov 02 02:09:14.057719 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:14.057692    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s"]
Nov 02 02:09:14.059624 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:09:14.059600    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051907ab_4e37_4298_9291_0df5fa65d735.slice/crio-7e4a69901c0ca39b1956b09c058511b041e605d30b8d1f4cfa3f8b9c4be67d03 WatchSource:0}: Error finding container 7e4a69901c0ca39b1956b09c058511b041e605d30b8d1f4cfa3f8b9c4be67d03: Status 404 returned error can't find the container with id 7e4a69901c0ca39b1956b09c058511b041e605d30b8d1f4cfa3f8b9c4be67d03
Nov 02 02:09:14.183667 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:14.183644    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-092079-9c6765d77-fxqv2"]
Nov 02 02:09:14.185519 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:09:14.185488    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ae28f8_e723_4bb3_b368_6fb9881d24bf.slice/crio-ed839cd145e72fc48f59945d576670745b633715b4c3c84953e2a210886312b3 WatchSource:0}: Error finding container ed839cd145e72fc48f59945d576670745b633715b4c3c84953e2a210886312b3: Status 404 returned error can't find the container with id ed839cd145e72fc48f59945d576670745b633715b4c3c84953e2a210886312b3
Nov 02 02:09:14.292134 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:14.292102    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-092079-9c6765d77-fxqv2" event={"ID":"c5ae28f8-e723-4bb3-b368-6fb9881d24bf","Type":"ContainerStarted","Data":"ef9a96fe1b24c4129f691c59adf53af988aea38b4ff3e03ee51d39c951b5c131"}
Nov 02 02:09:14.292321 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:14.292172    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-092079-9c6765d77-fxqv2"
Nov 02 02:09:14.292321 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:14.292199    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-092079-9c6765d77-fxqv2" event={"ID":"c5ae28f8-e723-4bb3-b368-6fb9881d24bf","Type":"ContainerStarted","Data":"ed839cd145e72fc48f59945d576670745b633715b4c3c84953e2a210886312b3"}
Nov 02 02:09:14.293927 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:14.293907    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s" event={"ID":"051907ab-4e37-4298-9291-0df5fa65d735","Type":"ContainerStarted","Data":"2b3cde8cadc4e23ff6dc01b40b8bb2bbe4bacf8fee07c566561afd211d1d3097"}
Nov 02 02:09:14.294001 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:14.293951    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s"
Nov 02 02:09:14.294001 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:14.293969    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s" event={"ID":"051907ab-4e37-4298-9291-0df5fa65d735","Type":"ContainerStarted","Data":"7e4a69901c0ca39b1956b09c058511b041e605d30b8d1f4cfa3f8b9c4be67d03"}
Nov 02 02:09:14.409717 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:14.409608    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-092079-9c6765d77-fxqv2" podStartSLOduration=1.409564024 podStartE2EDuration="1.409564024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:09:14.406154755 +0000 UTC m=+2201.847191267" watchObservedRunningTime="2026-11-02 02:09:14.409564024 +0000 UTC m=+2201.850600524"
Nov 02 02:09:14.531348 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:14.530695    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s" podStartSLOduration=1.530668757 podStartE2EDuration="1.530668757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:09:14.528532487 +0000 UTC m=+2201.969568985" watchObservedRunningTime="2026-11-02 02:09:14.530668757 +0000 UTC m=+2201.971705254"
Nov 02 02:09:15.849320 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:15.847213    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls"]
Nov 02 02:09:15.856399 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:15.855778    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls"
Nov 02 02:09:15.915235 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:15.915184    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls"]
Nov 02 02:09:15.915398 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:15.915196    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4211c0\""
Nov 02 02:09:15.977442 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:15.977409    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm"]
Nov 02 02:09:15.982266 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:15.982247    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm"
Nov 02 02:09:16.002429 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:16.002404    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad5288b9-b2da-41be-830a-72c877a15faf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls\" (UID: \"ad5288b9-b2da-41be-830a-72c877a15faf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls"
Nov 02 02:09:16.002584 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:16.002542    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqmq2\" (UniqueName: \"kubernetes.io/projected/ad5288b9-b2da-41be-830a-72c877a15faf-kube-api-access-jqmq2\") pod \"payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls\" (UID: \"ad5288b9-b2da-41be-830a-72c877a15faf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls"
Nov 02 02:09:16.035492 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:16.035459    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4211c0-dockercfg-jfhkz\""
Nov 02 02:09:16.039063 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:16.039032    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm"]
Nov 02 02:09:16.103784 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:16.103694    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmq2\" (UniqueName: \"kubernetes.io/projected/ad5288b9-b2da-41be-830a-72c877a15faf-kube-api-access-jqmq2\") pod \"payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls\" (UID: \"ad5288b9-b2da-41be-830a-72c877a15faf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls"
Nov 02 02:09:16.103784 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:16.103740    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cggdj\" (UniqueName: \"kubernetes.io/projected/0fa3f0ef-b33a-4fda-9e31-5393367463e2-kube-api-access-cggdj\") pod \"payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm\" (UID: \"0fa3f0ef-b33a-4fda-9e31-5393367463e2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm"
Nov 02 02:09:16.103784 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:16.103768    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa3f0ef-b33a-4fda-9e31-5393367463e2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm\" (UID: \"0fa3f0ef-b33a-4fda-9e31-5393367463e2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm"
Nov 02 02:09:16.104077 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:16.104004    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad5288b9-b2da-41be-830a-72c877a15faf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls\" (UID: \"ad5288b9-b2da-41be-830a-72c877a15faf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls"
Nov 02 02:09:16.104783 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:16.104763    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad5288b9-b2da-41be-830a-72c877a15faf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls\" (UID: \"ad5288b9-b2da-41be-830a-72c877a15faf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls"
Nov 02 02:09:16.164261 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:16.164212    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqmq2\" (UniqueName: \"kubernetes.io/projected/ad5288b9-b2da-41be-830a-72c877a15faf-kube-api-access-jqmq2\") pod \"payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls\" (UID: \"ad5288b9-b2da-41be-830a-72c877a15faf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls"
Nov 02 02:09:16.180682 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:16.180658    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls"
Nov 02 02:09:16.205050 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:16.205015    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cggdj\" (UniqueName: \"kubernetes.io/projected/0fa3f0ef-b33a-4fda-9e31-5393367463e2-kube-api-access-cggdj\") pod \"payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm\" (UID: \"0fa3f0ef-b33a-4fda-9e31-5393367463e2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm"
Nov 02 02:09:16.205150 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:16.205068    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa3f0ef-b33a-4fda-9e31-5393367463e2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm\" (UID: \"0fa3f0ef-b33a-4fda-9e31-5393367463e2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm"
Nov 02 02:09:16.206036 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:16.206013    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa3f0ef-b33a-4fda-9e31-5393367463e2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm\" (UID: \"0fa3f0ef-b33a-4fda-9e31-5393367463e2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm"
Nov 02 02:09:16.265781 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:16.265737    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cggdj\" (UniqueName: \"kubernetes.io/projected/0fa3f0ef-b33a-4fda-9e31-5393367463e2-kube-api-access-cggdj\") pod \"payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm\" (UID: \"0fa3f0ef-b33a-4fda-9e31-5393367463e2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm"
Nov 02 02:09:16.295807 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:16.295778    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm"
Nov 02 02:09:16.555023 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:16.554992    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls"]
Nov 02 02:09:16.556720 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:09:16.556691    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5288b9_b2da_41be_830a_72c877a15faf.slice/crio-5433587b5a8f79178b8fb750da8b81bdd38417908b9644fa2d844742ae623f20 WatchSource:0}: Error finding container 5433587b5a8f79178b8fb750da8b81bdd38417908b9644fa2d844742ae623f20: Status 404 returned error can't find the container with id 5433587b5a8f79178b8fb750da8b81bdd38417908b9644fa2d844742ae623f20
Nov 02 02:09:16.678519 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:16.678472    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm"]
Nov 02 02:09:16.681127 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:09:16.681101    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa3f0ef_b33a_4fda_9e31_5393367463e2.slice/crio-ee222f14d0d517b43879cef5a381338f2456cb80d75762138e1dfe9a7436838a WatchSource:0}: Error finding container ee222f14d0d517b43879cef5a381338f2456cb80d75762138e1dfe9a7436838a: Status 404 returned error can't find the container with id ee222f14d0d517b43879cef5a381338f2456cb80d75762138e1dfe9a7436838a
Nov 02 02:09:17.314643 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:17.314597    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm" event={"ID":"0fa3f0ef-b33a-4fda-9e31-5393367463e2","Type":"ContainerStarted","Data":"0a8b7a7a2adfb721a57a171e3f36dff77027dc44d026150974e57e36da92d08b"}
Nov 02 02:09:17.315142 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:17.314659    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm" event={"ID":"0fa3f0ef-b33a-4fda-9e31-5393367463e2","Type":"ContainerStarted","Data":"ee222f14d0d517b43879cef5a381338f2456cb80d75762138e1dfe9a7436838a"}
Nov 02 02:09:17.315142 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:17.314697    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm"
Nov 02 02:09:17.317880 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:17.317844    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls" event={"ID":"ad5288b9-b2da-41be-830a-72c877a15faf","Type":"ContainerStarted","Data":"0f2d5a6bbcaf74c9380464df41b94313328f40d990b29bc04310c5a73cbf58d3"}
Nov 02 02:09:17.318013 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:17.317942    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls"
Nov 02 02:09:17.318013 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:17.317995    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls" event={"ID":"ad5288b9-b2da-41be-830a-72c877a15faf","Type":"ContainerStarted","Data":"5433587b5a8f79178b8fb750da8b81bdd38417908b9644fa2d844742ae623f20"}
Nov 02 02:09:17.429863 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:17.429807    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm" podStartSLOduration=2.429790961 podStartE2EDuration="2.429790961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:09:17.4297389 +0000 UTC m=+2204.870775400" watchObservedRunningTime="2026-11-02 02:09:17.429790961 +0000 UTC m=+2204.870827458"
Nov 02 02:09:17.567438 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:17.566619    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls" podStartSLOduration=2.566595701 podStartE2EDuration="2.566595701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:09:17.548612272 +0000 UTC m=+2204.989648769" watchObservedRunningTime="2026-11-02 02:09:17.566595701 +0000 UTC m=+2205.007632200"
Nov 02 02:09:18.929370 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:18.929331    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh"]
Nov 02 02:09:23.929954 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:23.929908    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh" podUID="e89df719-7936-4331-980a-f2cbdfc0a8a4" containerName="payload-processing" containerID="cri-o://436b84b98829467bf07c102fc2c0a8f3917744faada743cf5c086319571833c6" gracePeriod=25
Nov 02 02:09:24.934630 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:24.934590    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh" podUID="e89df719-7936-4331-980a-f2cbdfc0a8a4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:09:25.302978 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:25.302949    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-092079-9c6765d77-fxqv2"
Nov 02 02:09:25.303128 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:25.303018    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s"
Nov 02 02:09:26.747545 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:26.747492    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb" podUID="c9628178-6287-4ba3-9467-93f84ae883ee" containerName="main" containerID="cri-o://aa26d60988af4c46d1e6754d72c47c81c28f6a19719c661dce559007c3422cd9" gracePeriod=45
Nov 02 02:09:27.008611 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.008553    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 02:09:27.068296 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.068267    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-home\" (UniqueName: \"kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-home\") pod \"c9628178-6287-4ba3-9467-93f84ae883ee\" (UID: \"c9628178-6287-4ba3-9467-93f84ae883ee\") "
Nov 02 02:09:27.068438 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.068334    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9628178-6287-4ba3-9467-93f84ae883ee-kube-api-access-tk9sv\" (UniqueName: \"kubernetes.io/projected/c9628178-6287-4ba3-9467-93f84ae883ee-kube-api-access-tk9sv\") pod \"c9628178-6287-4ba3-9467-93f84ae883ee\" (UID: \"c9628178-6287-4ba3-9467-93f84ae883ee\") "
Nov 02 02:09:27.068438 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.068363    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-model-cache\") pod \"c9628178-6287-4ba3-9467-93f84ae883ee\" (UID: \"c9628178-6287-4ba3-9467-93f84ae883ee\") "
Nov 02 02:09:27.068438 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.068392    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-tmp-dir\") pod \"c9628178-6287-4ba3-9467-93f84ae883ee\" (UID: \"c9628178-6287-4ba3-9467-93f84ae883ee\") "
Nov 02 02:09:27.068620 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.068486    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9628178-6287-4ba3-9467-93f84ae883ee-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9628178-6287-4ba3-9467-93f84ae883ee-tls-certs\") pod \"c9628178-6287-4ba3-9467-93f84ae883ee\" (UID: \"c9628178-6287-4ba3-9467-93f84ae883ee\") "
Nov 02 02:09:27.068620 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.068540    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-dshm\") pod \"c9628178-6287-4ba3-9467-93f84ae883ee\" (UID: \"c9628178-6287-4ba3-9467-93f84ae883ee\") "
Nov 02 02:09:27.068620 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.068547    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-home" pod "c9628178-6287-4ba3-9467-93f84ae883ee" (UID: "c9628178-6287-4ba3-9467-93f84ae883ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:27.068774 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.068656    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-model-cache" pod "c9628178-6287-4ba3-9467-93f84ae883ee" (UID: "c9628178-6287-4ba3-9467-93f84ae883ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:27.068774 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.068699    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-tmp-dir" pod "c9628178-6287-4ba3-9467-93f84ae883ee" (UID: "c9628178-6287-4ba3-9467-93f84ae883ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:27.068998 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.068978    5875 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-home\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:27.068998 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.068997    5875 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-model-cache\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:27.069369 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.069006    5875 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-tmp-dir\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:27.070761 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.070738    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-dshm" pod "c9628178-6287-4ba3-9467-93f84ae883ee" (UID: "c9628178-6287-4ba3-9467-93f84ae883ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:27.070761 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.070747    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9628178-6287-4ba3-9467-93f84ae883ee-kube-api-access-tk9sv" pod "c9628178-6287-4ba3-9467-93f84ae883ee" (UID: "c9628178-6287-4ba3-9467-93f84ae883ee"). InnerVolumeSpecName "kube-api-access-tk9sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:27.070761 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.070753    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9628178-6287-4ba3-9467-93f84ae883ee-tls-certs" pod "c9628178-6287-4ba3-9467-93f84ae883ee" (UID: "c9628178-6287-4ba3-9467-93f84ae883ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:09:27.170706 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.170668    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk9sv\" (UniqueName: \"kubernetes.io/projected/c9628178-6287-4ba3-9467-93f84ae883ee-kube-api-access-tk9sv\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:27.170706 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.170708    5875 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9628178-6287-4ba3-9467-93f84ae883ee-tls-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:27.170914 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.170724    5875 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9628178-6287-4ba3-9467-93f84ae883ee-dshm\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:27.429716 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.429622    5875 generic.go:356] "Generic (PLEG): container finished" podID="c9628178-6287-4ba3-9467-93f84ae883ee" containerID="aa26d60988af4c46d1e6754d72c47c81c28f6a19719c661dce559007c3422cd9" exitCode=0
Nov 02 02:09:27.429716 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.429684    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"
Nov 02 02:09:27.429929 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.429708    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb" event={"ID":"c9628178-6287-4ba3-9467-93f84ae883ee","Type":"ContainerDied","Data":"aa26d60988af4c46d1e6754d72c47c81c28f6a19719c661dce559007c3422cd9"}
Nov 02 02:09:27.429929 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.429756    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb" event={"ID":"c9628178-6287-4ba3-9467-93f84ae883ee","Type":"ContainerDied","Data":"afb681e048511ef52f61a035dfee90c86ce83e8439470a564f11cc735cbcf262"}
Nov 02 02:09:27.429929 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.429772    5875 scope.go:122] "RemoveContainer" containerID="aa26d60988af4c46d1e6754d72c47c81c28f6a19719c661dce559007c3422cd9"
Nov 02 02:09:27.440186 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.440170    5875 scope.go:122] "RemoveContainer" containerID="aa26d60988af4c46d1e6754d72c47c81c28f6a19719c661dce559007c3422cd9"
Nov 02 02:09:27.440515 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:09:27.440491    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa26d60988af4c46d1e6754d72c47c81c28f6a19719c661dce559007c3422cd9\": container with ID starting with aa26d60988af4c46d1e6754d72c47c81c28f6a19719c661dce559007c3422cd9 not found: ID does not exist" containerID="aa26d60988af4c46d1e6754d72c47c81c28f6a19719c661dce559007c3422cd9"
Nov 02 02:09:27.440598 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.440523    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa26d60988af4c46d1e6754d72c47c81c28f6a19719c661dce559007c3422cd9"} err="failed to get container status \"aa26d60988af4c46d1e6754d72c47c81c28f6a19719c661dce559007c3422cd9\": rpc error: code = NotFound desc = could not find container \"aa26d60988af4c46d1e6754d72c47c81c28f6a19719c661dce559007c3422cd9\": container with ID starting with aa26d60988af4c46d1e6754d72c47c81c28f6a19719c661dce559007c3422cd9 not found: ID does not exist"
Nov 02 02:09:27.594929 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.594893    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"]
Nov 02 02:09:27.644847 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.644816    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb"]
Nov 02 02:09:27.947854 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:27.947817    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-061fab/premium-w3-061fab-kserve-7bb7d67d85-92kbb" podUID="c9628178-6287-4ba3-9467-93f84ae883ee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:09:28.339597 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:28.339571    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls"
Nov 02 02:09:28.339890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:28.339635    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm"
Nov 02 02:09:28.395907 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:28.395873    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"]
Nov 02 02:09:28.396129 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:28.396109    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg" podUID="f87d2fb1-742f-4632-bd55-5710c177b1f6" containerName="istio-proxy" containerID="cri-o://ad335955b578b7b7cbdf96675771b01a6ec17c90aa13782b3c6118b004819c28" gracePeriod=30
Nov 02 02:09:28.696161 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:28.696086    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9628178-6287-4ba3-9467-93f84ae883ee" path="/var/lib/kubelet/pods/c9628178-6287-4ba3-9467-93f84ae883ee/volumes"
Nov 02 02:09:33.689576 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.689547    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 02:09:33.756800 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.756761    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-token\" (UniqueName: \"kubernetes.io/projected/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-token\") pod \"f87d2fb1-742f-4632-bd55-5710c177b1f6\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") "
Nov 02 02:09:33.757177 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.756859    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-workload-certs\") pod \"f87d2fb1-742f-4632-bd55-5710c177b1f6\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") "
Nov 02 02:09:33.757177 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.756916    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-workload-socket\") pod \"f87d2fb1-742f-4632-bd55-5710c177b1f6\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") "
Nov 02 02:09:33.757177 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.757029    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-credential-socket\") pod \"f87d2fb1-742f-4632-bd55-5710c177b1f6\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") "
Nov 02 02:09:33.757177 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.757102    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-envoy\") pod \"f87d2fb1-742f-4632-bd55-5710c177b1f6\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") "
Nov 02 02:09:33.757177 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.757169    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f87d2fb1-742f-4632-bd55-5710c177b1f6-kube-api-access-24qkj\" (UniqueName: \"kubernetes.io/projected/f87d2fb1-742f-4632-bd55-5710c177b1f6-kube-api-access-24qkj\") pod \"f87d2fb1-742f-4632-bd55-5710c177b1f6\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") "
Nov 02 02:09:33.757521 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.757260    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-podinfo\") pod \"f87d2fb1-742f-4632-bd55-5710c177b1f6\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") "
Nov 02 02:09:33.757521 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.757319    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-workload-certs" pod "f87d2fb1-742f-4632-bd55-5710c177b1f6" (UID: "f87d2fb1-742f-4632-bd55-5710c177b1f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:33.757521 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.757343    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f87d2fb1-742f-4632-bd55-5710c177b1f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f87d2fb1-742f-4632-bd55-5710c177b1f6-istiod-ca-cert\") pod \"f87d2fb1-742f-4632-bd55-5710c177b1f6\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") "
Nov 02 02:09:33.757773 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.757722    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f87d2fb1-742f-4632-bd55-5710c177b1f6-istiod-ca-cert" pod "f87d2fb1-742f-4632-bd55-5710c177b1f6" (UID: "f87d2fb1-742f-4632-bd55-5710c177b1f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:09:33.757912 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.757816    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-data\") pod \"f87d2fb1-742f-4632-bd55-5710c177b1f6\" (UID: \"f87d2fb1-742f-4632-bd55-5710c177b1f6\") "
Nov 02 02:09:33.758331 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.758301    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-workload-socket" pod "f87d2fb1-742f-4632-bd55-5710c177b1f6" (UID: "f87d2fb1-742f-4632-bd55-5710c177b1f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:33.758429 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.758371    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-data" pod "f87d2fb1-742f-4632-bd55-5710c177b1f6" (UID: "f87d2fb1-742f-4632-bd55-5710c177b1f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:33.758608 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.758587    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:33.758608 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.758615    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:33.758770 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.758630    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f87d2fb1-742f-4632-bd55-5710c177b1f6-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:33.758770 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.758646    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:33.758770 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.758591    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-credential-socket" pod "f87d2fb1-742f-4632-bd55-5710c177b1f6" (UID: "f87d2fb1-742f-4632-bd55-5710c177b1f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:33.759648 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.759627    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-envoy" pod "f87d2fb1-742f-4632-bd55-5710c177b1f6" (UID: "f87d2fb1-742f-4632-bd55-5710c177b1f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:09:33.759757 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.759654    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-token" pod "f87d2fb1-742f-4632-bd55-5710c177b1f6" (UID: "f87d2fb1-742f-4632-bd55-5710c177b1f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:33.759873 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.759850    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-podinfo" pod "f87d2fb1-742f-4632-bd55-5710c177b1f6" (UID: "f87d2fb1-742f-4632-bd55-5710c177b1f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:09:33.759873 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.759860    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f87d2fb1-742f-4632-bd55-5710c177b1f6-kube-api-access-24qkj" pod "f87d2fb1-742f-4632-bd55-5710c177b1f6" (UID: "f87d2fb1-742f-4632-bd55-5710c177b1f6"). InnerVolumeSpecName "kube-api-access-24qkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:33.860638 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.860589    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:33.860638 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.860626    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:33.860638 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.860641    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:33.860875 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.860655    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24qkj\" (UniqueName: \"kubernetes.io/projected/f87d2fb1-742f-4632-bd55-5710c177b1f6-kube-api-access-24qkj\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:33.860875 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:33.860670    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f87d2fb1-742f-4632-bd55-5710c177b1f6-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:34.475791 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:34.475759    5875 generic.go:356] "Generic (PLEG): container finished" podID="f87d2fb1-742f-4632-bd55-5710c177b1f6" containerID="ad335955b578b7b7cbdf96675771b01a6ec17c90aa13782b3c6118b004819c28" exitCode=0
Nov 02 02:09:34.476016 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:34.475839    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"
Nov 02 02:09:34.476016 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:34.475854    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg" event={"ID":"f87d2fb1-742f-4632-bd55-5710c177b1f6","Type":"ContainerDied","Data":"ad335955b578b7b7cbdf96675771b01a6ec17c90aa13782b3c6118b004819c28"}
Nov 02 02:09:34.476016 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:34.475910    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg" event={"ID":"f87d2fb1-742f-4632-bd55-5710c177b1f6","Type":"ContainerDied","Data":"d0d21d9448b2580f23abe0184fff9e823999866af46735114f9530d01991506a"}
Nov 02 02:09:34.476016 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:34.475932    5875 scope.go:122] "RemoveContainer" containerID="ad335955b578b7b7cbdf96675771b01a6ec17c90aa13782b3c6118b004819c28"
Nov 02 02:09:34.486450 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:34.486433    5875 scope.go:122] "RemoveContainer" containerID="ad335955b578b7b7cbdf96675771b01a6ec17c90aa13782b3c6118b004819c28"
Nov 02 02:09:34.486748 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:09:34.486731    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad335955b578b7b7cbdf96675771b01a6ec17c90aa13782b3c6118b004819c28\": container with ID starting with ad335955b578b7b7cbdf96675771b01a6ec17c90aa13782b3c6118b004819c28 not found: ID does not exist" containerID="ad335955b578b7b7cbdf96675771b01a6ec17c90aa13782b3c6118b004819c28"
Nov 02 02:09:34.486811 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:34.486757    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad335955b578b7b7cbdf96675771b01a6ec17c90aa13782b3c6118b004819c28"} err="failed to get container status \"ad335955b578b7b7cbdf96675771b01a6ec17c90aa13782b3c6118b004819c28\": rpc error: code = NotFound desc = could not find container \"ad335955b578b7b7cbdf96675771b01a6ec17c90aa13782b3c6118b004819c28\": container with ID starting with ad335955b578b7b7cbdf96675771b01a6ec17c90aa13782b3c6118b004819c28 not found: ID does not exist"
Nov 02 02:09:34.697018 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:34.696991    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"]
Nov 02 02:09:34.738659 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:34.738584    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ed0600-openshift-default-55768f8df5-ntmtg"]
Nov 02 02:09:34.934629 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:34.934598    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh" podUID="e89df719-7936-4331-980a-f2cbdfc0a8a4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:09:35.304679 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:35.304651    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:09:36.695173 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:36.695140    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f87d2fb1-742f-4632-bd55-5710c177b1f6" path="/var/lib/kubelet/pods/f87d2fb1-742f-4632-bd55-5710c177b1f6/volumes"
Nov 02 02:09:41.317810 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:09:41.317775    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:09:44.197445 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:44.197384    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh"
Nov 02 02:09:44.207470 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:44.207446    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e89df719-7936-4331-980a-f2cbdfc0a8a4-kube-api-access-b7n7w\" (UniqueName: \"kubernetes.io/projected/e89df719-7936-4331-980a-f2cbdfc0a8a4-kube-api-access-b7n7w\") pod \"e89df719-7936-4331-980a-f2cbdfc0a8a4\" (UID: \"e89df719-7936-4331-980a-f2cbdfc0a8a4\") "
Nov 02 02:09:44.207612 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:44.207573    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e89df719-7936-4331-980a-f2cbdfc0a8a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e89df719-7936-4331-980a-f2cbdfc0a8a4-plugins-config-volume\") pod \"e89df719-7936-4331-980a-f2cbdfc0a8a4\" (UID: \"e89df719-7936-4331-980a-f2cbdfc0a8a4\") "
Nov 02 02:09:44.207911 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:44.207889    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89df719-7936-4331-980a-f2cbdfc0a8a4-plugins-config-volume" pod "e89df719-7936-4331-980a-f2cbdfc0a8a4" (UID: "e89df719-7936-4331-980a-f2cbdfc0a8a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:09:44.209696 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:44.209674    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89df719-7936-4331-980a-f2cbdfc0a8a4-kube-api-access-b7n7w" pod "e89df719-7936-4331-980a-f2cbdfc0a8a4" (UID: "e89df719-7936-4331-980a-f2cbdfc0a8a4"). InnerVolumeSpecName "kube-api-access-b7n7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:09:44.308797 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:44.308767    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e89df719-7936-4331-980a-f2cbdfc0a8a4-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:44.308797 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:44.308792    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7n7w\" (UniqueName: \"kubernetes.io/projected/e89df719-7936-4331-980a-f2cbdfc0a8a4-kube-api-access-b7n7w\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:09:44.567045 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:44.567015    5875 generic.go:356] "Generic (PLEG): container finished" podID="e89df719-7936-4331-980a-f2cbdfc0a8a4" containerID="436b84b98829467bf07c102fc2c0a8f3917744faada743cf5c086319571833c6" exitCode=0
Nov 02 02:09:44.567255 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:44.567059    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh"
Nov 02 02:09:44.567255 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:44.567063    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh" event={"ID":"e89df719-7936-4331-980a-f2cbdfc0a8a4","Type":"ContainerDied","Data":"436b84b98829467bf07c102fc2c0a8f3917744faada743cf5c086319571833c6"}
Nov 02 02:09:44.567255 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:44.567206    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh" event={"ID":"e89df719-7936-4331-980a-f2cbdfc0a8a4","Type":"ContainerDied","Data":"d21e713f0ec91d68150a9ed597c81e67406bb3f8c0546503c604352a218e1d18"}
Nov 02 02:09:44.567255 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:44.567248    5875 scope.go:122] "RemoveContainer" containerID="436b84b98829467bf07c102fc2c0a8f3917744faada743cf5c086319571833c6"
Nov 02 02:09:44.576969 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:44.576953    5875 scope.go:122] "RemoveContainer" containerID="436b84b98829467bf07c102fc2c0a8f3917744faada743cf5c086319571833c6"
Nov 02 02:09:44.577268 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:09:44.577250    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"436b84b98829467bf07c102fc2c0a8f3917744faada743cf5c086319571833c6\": container with ID starting with 436b84b98829467bf07c102fc2c0a8f3917744faada743cf5c086319571833c6 not found: ID does not exist" containerID="436b84b98829467bf07c102fc2c0a8f3917744faada743cf5c086319571833c6"
Nov 02 02:09:44.577326 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:44.577276    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"436b84b98829467bf07c102fc2c0a8f3917744faada743cf5c086319571833c6"} err="failed to get container status \"436b84b98829467bf07c102fc2c0a8f3917744faada743cf5c086319571833c6\": rpc error: code = NotFound desc = could not find container \"436b84b98829467bf07c102fc2c0a8f3917744faada743cf5c086319571833c6\": container with ID starting with 436b84b98829467bf07c102fc2c0a8f3917744faada743cf5c086319571833c6 not found: ID does not exist"
Nov 02 02:09:44.730715 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:44.730683    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh"]
Nov 02 02:09:44.734936 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:44.734911    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-061fab-6bfd68d6cc-tczxh"]
Nov 02 02:09:46.694853 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:46.694820    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e89df719-7936-4331-980a-f2cbdfc0a8a4" path="/var/lib/kubelet/pods/e89df719-7936-4331-980a-f2cbdfc0a8a4/volumes"
Nov 02 02:09:47.408338 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.408305    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"]
Nov 02 02:09:47.408883 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.408870    5875 cpu_manager.go:414] "containerMap: removing container" podUID="e89df719-7936-4331-980a-f2cbdfc0a8a4" containerName="payload-processing"
Nov 02 02:09:47.408938 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.408884    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89df719-7936-4331-980a-f2cbdfc0a8a4" containerName="payload-processing"
Nov 02 02:09:47.408938 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.408907    5875 cpu_manager.go:414] "containerMap: removing container" podUID="f87d2fb1-742f-4632-bd55-5710c177b1f6" containerName="istio-proxy"
Nov 02 02:09:47.408938 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.408912    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f87d2fb1-742f-4632-bd55-5710c177b1f6" containerName="istio-proxy"
Nov 02 02:09:47.408938 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.408930    5875 cpu_manager.go:414] "containerMap: removing container" podUID="c9628178-6287-4ba3-9467-93f84ae883ee" containerName="main"
Nov 02 02:09:47.408938 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.408935    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9628178-6287-4ba3-9467-93f84ae883ee" containerName="main"
Nov 02 02:09:47.409128 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.409014    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="e89df719-7936-4331-980a-f2cbdfc0a8a4" containerName="payload-processing"
Nov 02 02:09:47.409128 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.409026    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="f87d2fb1-742f-4632-bd55-5710c177b1f6" containerName="istio-proxy"
Nov 02 02:09:47.409128 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.409034    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9628178-6287-4ba3-9467-93f84ae883ee" containerName="main"
Nov 02 02:09:47.413895 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.413875    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.443734 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.443688    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqwtg\" (UniqueName: \"kubernetes.io/projected/846f35bf-08b7-4447-b7f5-47602b851bd5-kube-api-access-zqwtg\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.443902 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.443810    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.443902 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.443862    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.443902 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.443890    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.444084 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.443961    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.444084 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.443998    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.444084 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.444044    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/846f35bf-08b7-4447-b7f5-47602b851bd5-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.444301 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.444097    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.444301 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.444145    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.458175 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.458145    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"]
Nov 02 02:09:47.462192 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.462166    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.470524 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.470504    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-55bc84-openshift-default-dockercfg-7g4tr\""
Nov 02 02:09:47.519555 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.519521    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"]
Nov 02 02:09:47.525878 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.525852    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"]
Nov 02 02:09:47.545823 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.545792    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.545977 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.545834    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqwtg\" (UniqueName: \"kubernetes.io/projected/846f35bf-08b7-4447-b7f5-47602b851bd5-kube-api-access-zqwtg\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.546041 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.545965    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.546041 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.546029    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.546155 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.546105    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.546210 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.546157    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.546210 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.546182    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.546405 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.546245    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.546405 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.546284    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.546405 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.546351    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.546405 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.546385    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.546624 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.546417    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.546624 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.546457    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/846f35bf-08b7-4447-b7f5-47602b851bd5-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.546624 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.546507    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdblr\" (UniqueName: \"kubernetes.io/projected/883a5d43-c155-47d2-bcc0-29a2588b805a-kube-api-access-gdblr\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.546624 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.546539    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.546624 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.546588    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.546624 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.546595    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.546624 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.546619    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.546956 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.546646    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/883a5d43-c155-47d2-bcc0-29a2588b805a-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.547239 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.547198    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.547396 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.547372    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.547503 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.547477    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.547598 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.547572    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/846f35bf-08b7-4447-b7f5-47602b851bd5-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.549152 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.549125    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.550485 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.550457    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.607813 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.607778    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqwtg\" (UniqueName: \"kubernetes.io/projected/846f35bf-08b7-4447-b7f5-47602b851bd5-kube-api-access-zqwtg\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.607993 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.607962    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.647728 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.647694    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.647864 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.647742    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.647864 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.647808    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.647864 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.647841    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.648017 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.647910    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.648017 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.647968    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdblr\" (UniqueName: \"kubernetes.io/projected/883a5d43-c155-47d2-bcc0-29a2588b805a-kube-api-access-gdblr\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.648130 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.648019    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.648130 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.648049    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/883a5d43-c155-47d2-bcc0-29a2588b805a-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.648130 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.648110    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.648318 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.648129    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-workload-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.648418 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.648383    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-workload-certs\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.648510 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.648488    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-data\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.648572 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.648562    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-credential-socket\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.648870 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.648781    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/883a5d43-c155-47d2-bcc0-29a2588b805a-istiod-ca-cert\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.650489 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.650470    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-envoy\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.650658 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.650635    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-podinfo\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.712048 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.712013    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-token\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.712432 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.712147    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdblr\" (UniqueName: \"kubernetes.io/projected/883a5d43-c155-47d2-bcc0-29a2588b805a-kube-api-access-gdblr\") pod \"e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") " pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:47.728141 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.728121    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:47.776421 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:47.776391    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:48.160824 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:48.160744    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"]
Nov 02 02:09:48.198153 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:09:48.198120    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846f35bf_08b7_4447_b7f5_47602b851bd5.slice/crio-f2d76d9a35b5a27acf5324fc155551f412a596fe44707ffa49cd0ada33cc7797 WatchSource:0}: Error finding container f2d76d9a35b5a27acf5324fc155551f412a596fe44707ffa49cd0ada33cc7797: Status 404 returned error can't find the container with id f2d76d9a35b5a27acf5324fc155551f412a596fe44707ffa49cd0ada33cc7797
Nov 02 02:09:48.200786 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:48.200752    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:09:48.200886 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:48.200821    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:09:48.200886 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:48.200851    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:09:48.212874 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:48.212849    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"]
Nov 02 02:09:48.214847 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:09:48.214819    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883a5d43_c155_47d2_bcc0_29a2588b805a.slice/crio-ab09cad5bb122042a3b72826e1c65b57a5c2420c0707f8718bebbf302fe145c2 WatchSource:0}: Error finding container ab09cad5bb122042a3b72826e1c65b57a5c2420c0707f8718bebbf302fe145c2: Status 404 returned error can't find the container with id ab09cad5bb122042a3b72826e1c65b57a5c2420c0707f8718bebbf302fe145c2
Nov 02 02:09:48.217385 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:48.217355    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:09:48.217483 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:48.217417    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:09:48.217483 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:48.217444    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:09:48.595517 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:48.595483    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9" event={"ID":"846f35bf-08b7-4447-b7f5-47602b851bd5","Type":"ContainerStarted","Data":"7d53b1f53020ad0873dc95b099a0c4e80bdddc6db0ebb1a9c0ae8343aafcbe43"}
Nov 02 02:09:48.595709 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:48.595543    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9" event={"ID":"846f35bf-08b7-4447-b7f5-47602b851bd5","Type":"ContainerStarted","Data":"f2d76d9a35b5a27acf5324fc155551f412a596fe44707ffa49cd0ada33cc7797"}
Nov 02 02:09:48.597403 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:48.597378    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95" event={"ID":"883a5d43-c155-47d2-bcc0-29a2588b805a","Type":"ContainerStarted","Data":"e977dbe1afbe67b9451231612e20f58866314b8d374faa4ad03be04597fe2e5f"}
Nov 02 02:09:48.597528 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:48.597429    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95" event={"ID":"883a5d43-c155-47d2-bcc0-29a2588b805a","Type":"ContainerStarted","Data":"ab09cad5bb122042a3b72826e1c65b57a5c2420c0707f8718bebbf302fe145c2"}
Nov 02 02:09:48.719382 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:48.719323    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9" podStartSLOduration=1.719305509 podStartE2EDuration="1.719305509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:09:48.7155983 +0000 UTC m=+2236.156634798" watchObservedRunningTime="2026-11-02 02:09:48.719305509 +0000 UTC m=+2236.160342008"
Nov 02 02:09:48.728993 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:48.728967    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:48.777506 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:48.777480    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:09:48.836984 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:48.836926    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95" podStartSLOduration=1.836908349 podStartE2EDuration="1.836908349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:09:48.834764227 +0000 UTC m=+2236.275800726" watchObservedRunningTime="2026-11-02 02:09:48.836908349 +0000 UTC m=+2236.277944847"
Nov 02 02:09:49.742400 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.742364    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 02 02:09:49.743105 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.743067    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9" podUID="846f35bf-08b7-4447-b7f5-47602b851bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 02 02:09:49.772696 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.772654    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"]
Nov 02 02:09:49.777474 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.777454    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:49.778421 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.778397    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 02 02:09:49.778522 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.778454    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95" podUID="883a5d43-c155-47d2-bcc0-29a2588b805a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 02 02:09:49.848083 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.848046    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"]
Nov 02 02:09:49.853574 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.853537    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.860252 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.858657    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"]
Nov 02 02:09:49.872022 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.871996    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"]
Nov 02 02:09:49.872165 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.872010    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:49.881548 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.881516    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2d802d-8fbd-430d-ae72-29b0e217714d-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:49.881698 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.881682    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a50af6e0-a390-47d4-a4d2-7f90a3055661-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.881787 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.881764    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2d802d-8fbd-430d-ae72-29b0e217714d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:49.881884 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.881868    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a50af6e0-a390-47d4-a4d2-7f90a3055661-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.881966 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.881915    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2d802d-8fbd-430d-ae72-29b0e217714d-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:49.882053 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.882007    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a50af6e0-a390-47d4-a4d2-7f90a3055661-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.882053 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.882040    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4qcj\" (UniqueName: \"kubernetes.io/projected/0c2d802d-8fbd-430d-ae72-29b0e217714d-kube-api-access-f4qcj\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:49.882210 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.882073    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50af6e0-a390-47d4-a4d2-7f90a3055661-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.882210 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.882099    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d802d-8fbd-430d-ae72-29b0e217714d-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:49.882210 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.882183    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtnrc\" (UniqueName: \"kubernetes.io/projected/a50af6e0-a390-47d4-a4d2-7f90a3055661-kube-api-access-wtnrc\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.882452 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.882258    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2d802d-8fbd-430d-ae72-29b0e217714d-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:49.882452 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.882294    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2d802d-8fbd-430d-ae72-29b0e217714d-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:49.882452 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.882357    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2d802d-8fbd-430d-ae72-29b0e217714d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:49.882452 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.882388    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a50af6e0-a390-47d4-a4d2-7f90a3055661-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.882452 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.882417    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d802d-8fbd-430d-ae72-29b0e217714d-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:49.882452 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.882448    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a50af6e0-a390-47d4-a4d2-7f90a3055661-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.882776 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.882493    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50af6e0-a390-47d4-a4d2-7f90a3055661-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.882776 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.882527    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a50af6e0-a390-47d4-a4d2-7f90a3055661-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.888353 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.886416    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"]
Nov 02 02:09:49.918245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.896401    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"]
Nov 02 02:09:49.983327 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.983299    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba032348-8064-40c7-8278-0115c943882e-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:49.983530 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.983341    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2d802d-8fbd-430d-ae72-29b0e217714d-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:49.983530 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.983412    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a50af6e0-a390-47d4-a4d2-7f90a3055661-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.983530 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.983448    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2d802d-8fbd-430d-ae72-29b0e217714d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:49.983530 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.983480    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a50af6e0-a390-47d4-a4d2-7f90a3055661-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.983530 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.983504    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2d802d-8fbd-430d-ae72-29b0e217714d-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:49.983530 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.983531    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba032348-8064-40c7-8278-0115c943882e-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:49.983871 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.983551    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba032348-8064-40c7-8278-0115c943882e-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:49.983871 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.983574    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a50af6e0-a390-47d4-a4d2-7f90a3055661-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.983871 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.983592    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4qcj\" (UniqueName: \"kubernetes.io/projected/0c2d802d-8fbd-430d-ae72-29b0e217714d-kube-api-access-f4qcj\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:49.983871 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.983613    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50af6e0-a390-47d4-a4d2-7f90a3055661-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.983871 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.983628    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d802d-8fbd-430d-ae72-29b0e217714d-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:49.983871 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.983651    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba032348-8064-40c7-8278-0115c943882e-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:49.983871 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.983679    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba032348-8064-40c7-8278-0115c943882e-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:49.983871 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.983721    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtnrc\" (UniqueName: \"kubernetes.io/projected/a50af6e0-a390-47d4-a4d2-7f90a3055661-kube-api-access-wtnrc\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.983871 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.983749    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba032348-8064-40c7-8278-0115c943882e-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:49.983871 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.983769    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2d802d-8fbd-430d-ae72-29b0e217714d-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:49.983871 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.983792    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2d802d-8fbd-430d-ae72-29b0e217714d-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:49.983871 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.983810    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrncs\" (UniqueName: \"kubernetes.io/projected/ba032348-8064-40c7-8278-0115c943882e-kube-api-access-jrncs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:49.983871 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.983834    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba032348-8064-40c7-8278-0115c943882e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:49.983871 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.983832    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2d802d-8fbd-430d-ae72-29b0e217714d-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:49.983871 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.983853    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2d802d-8fbd-430d-ae72-29b0e217714d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:49.984743 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.983895    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a50af6e0-a390-47d4-a4d2-7f90a3055661-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.984743 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.983948    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d802d-8fbd-430d-ae72-29b0e217714d-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:49.984743 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.983983    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba032348-8064-40c7-8278-0115c943882e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:49.984743 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.984012    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a50af6e0-a390-47d4-a4d2-7f90a3055661-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.984743 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.984061    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50af6e0-a390-47d4-a4d2-7f90a3055661-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.984743 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.984094    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a50af6e0-a390-47d4-a4d2-7f90a3055661-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.984743 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.984510    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a50af6e0-a390-47d4-a4d2-7f90a3055661-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.985114 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.984800    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d802d-8fbd-430d-ae72-29b0e217714d-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:49.985114 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.984774    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2d802d-8fbd-430d-ae72-29b0e217714d-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:49.985114 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.984879    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a50af6e0-a390-47d4-a4d2-7f90a3055661-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.985114 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.985101    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50af6e0-a390-47d4-a4d2-7f90a3055661-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.985382 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.985156    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d802d-8fbd-430d-ae72-29b0e217714d-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:49.985682 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.985633    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50af6e0-a390-47d4-a4d2-7f90a3055661-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.985821 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.985783    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2d802d-8fbd-430d-ae72-29b0e217714d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:49.985821 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.985805    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a50af6e0-a390-47d4-a4d2-7f90a3055661-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.986543 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.986521    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a50af6e0-a390-47d4-a4d2-7f90a3055661-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.986644 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.986592    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a50af6e0-a390-47d4-a4d2-7f90a3055661-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:49.987201 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.987184    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2d802d-8fbd-430d-ae72-29b0e217714d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:49.987486 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:49.987468    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2d802d-8fbd-430d-ae72-29b0e217714d-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:50.056041 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.055963    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtnrc\" (UniqueName: \"kubernetes.io/projected/a50af6e0-a390-47d4-a4d2-7f90a3055661-kube-api-access-wtnrc\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:50.056658 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.056632    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a50af6e0-a390-47d4-a4d2-7f90a3055661-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-727fm\" (UID: \"a50af6e0-a390-47d4-a4d2-7f90a3055661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:50.056836 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.056812    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2d802d-8fbd-430d-ae72-29b0e217714d-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:50.057067 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.057049    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4qcj\" (UniqueName: \"kubernetes.io/projected/0c2d802d-8fbd-430d-ae72-29b0e217714d-kube-api-access-f4qcj\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ghslr\" (UID: \"0c2d802d-8fbd-430d-ae72-29b0e217714d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:50.085095 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.085069    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba032348-8064-40c7-8278-0115c943882e-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:50.085284 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.085159    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba032348-8064-40c7-8278-0115c943882e-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:50.085284 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.085180    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba032348-8064-40c7-8278-0115c943882e-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:50.085284 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.085244    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba032348-8064-40c7-8278-0115c943882e-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:50.085284 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.085273    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba032348-8064-40c7-8278-0115c943882e-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:50.085463 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.085312    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba032348-8064-40c7-8278-0115c943882e-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:50.085463 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.085339    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrncs\" (UniqueName: \"kubernetes.io/projected/ba032348-8064-40c7-8278-0115c943882e-kube-api-access-jrncs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:50.085463 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.085361    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba032348-8064-40c7-8278-0115c943882e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:50.085463 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.085389    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba032348-8064-40c7-8278-0115c943882e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:50.085649 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.085479    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba032348-8064-40c7-8278-0115c943882e-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:50.085649 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.085514    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba032348-8064-40c7-8278-0115c943882e-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:50.085868 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.085852    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba032348-8064-40c7-8278-0115c943882e-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:50.085972 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.085955    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba032348-8064-40c7-8278-0115c943882e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:50.086298 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.086278    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba032348-8064-40c7-8278-0115c943882e-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:50.087471 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.087444    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba032348-8064-40c7-8278-0115c943882e-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:50.088792 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.088772    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba032348-8064-40c7-8278-0115c943882e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:50.092682 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.092654    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:50.160619 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.160397    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba032348-8064-40c7-8278-0115c943882e-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:50.164841 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.164814    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrncs\" (UniqueName: \"kubernetes.io/projected/ba032348-8064-40c7-8278-0115c943882e-kube-api-access-jrncs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-h8px6\" (UID: \"ba032348-8064-40c7-8278-0115c943882e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:50.179164 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.179141    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:50.192639 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.192602    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:50.655904 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.655804    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"]
Nov 02 02:09:50.685941 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:09:50.685916    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2d802d_8fbd_430d_ae72_29b0e217714d.slice/crio-07e90901701dc6bce0dde100de2a96863dbe6442e72f3b659b31c7c1a852508d WatchSource:0}: Error finding container 07e90901701dc6bce0dde100de2a96863dbe6442e72f3b659b31c7c1a852508d: Status 404 returned error can't find the container with id 07e90901701dc6bce0dde100de2a96863dbe6442e72f3b659b31c7c1a852508d
Nov 02 02:09:50.688463 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.688429    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:09:50.688540 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.688495    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:09:50.688540 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.688524    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:09:50.728780 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.728652    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 02 02:09:50.728780 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.728709    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9" podUID="846f35bf-08b7-4447-b7f5-47602b851bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 02 02:09:50.766243 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.755830    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"]
Nov 02 02:09:50.766243 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.761146    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"]
Nov 02 02:09:50.770570 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.769399    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:09:50.770570 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.769455    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:09:50.770570 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.769484    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:09:50.777469 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.774257    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:09:50.777469 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.774332    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:09:50.777469 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.774360    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:09:50.777469 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.777337    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 02 02:09:50.777469 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:50.777395    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95" podUID="883a5d43-c155-47d2-bcc0-29a2588b805a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 02 02:09:51.795799 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:51.795350    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 02 02:09:51.795799 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:51.795412    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95" podUID="883a5d43-c155-47d2-bcc0-29a2588b805a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 02 02:09:51.810458 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:51.810326    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 02 02:09:51.810458 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:51.810388    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9" podUID="846f35bf-08b7-4447-b7f5-47602b851bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 02 02:09:51.889434 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:51.888296    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6" event={"ID":"ba032348-8064-40c7-8278-0115c943882e","Type":"ContainerStarted","Data":"af96155644561c2004c78fdb4426f13be225bc0beac564e1acf80b4d803e2b71"}
Nov 02 02:09:51.889434 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:51.888366    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6" event={"ID":"ba032348-8064-40c7-8278-0115c943882e","Type":"ContainerStarted","Data":"994bb75402c380676080f51129d6a95dae6eddc549b95e06e93eb6d839761579"}
Nov 02 02:09:51.931269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:51.910305    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm" event={"ID":"a50af6e0-a390-47d4-a4d2-7f90a3055661","Type":"ContainerStarted","Data":"6170cce00758091d1337c9112bd3293e16e01ef0072d7e5efa12d61f5cf15915"}
Nov 02 02:09:51.931269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:51.910374    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm" event={"ID":"a50af6e0-a390-47d4-a4d2-7f90a3055661","Type":"ContainerStarted","Data":"305f1e3d3cd2b59b9a5197fa5e7aa34e1ea343a9948c7d346b70273a32cc41c7"}
Nov 02 02:09:51.942267 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:51.940506    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr" event={"ID":"0c2d802d-8fbd-430d-ae72-29b0e217714d","Type":"ContainerStarted","Data":"37cb46e1874873cffe0f445bdbcb737a25193e9cac1aa01555c15ee9380ee046"}
Nov 02 02:09:51.942267 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:51.940574    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr" event={"ID":"0c2d802d-8fbd-430d-ae72-29b0e217714d","Type":"ContainerStarted","Data":"07e90901701dc6bce0dde100de2a96863dbe6442e72f3b659b31c7c1a852508d"}
Nov 02 02:09:52.100307 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:52.098751    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:09:52.100307 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:52.098967    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ghslr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 02:09:52.100307 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:52.099026    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr" podUID="0c2d802d-8fbd-430d-ae72-29b0e217714d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 02:09:52.161144 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:52.161077    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6" podStartSLOduration=3.161054453 podStartE2EDuration="3.161054453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:09:52.017087524 +0000 UTC m=+2239.458124022" watchObservedRunningTime="2026-11-02 02:09:52.161054453 +0000 UTC m=+2239.602090950"
Nov 02 02:09:52.188273 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:52.188240    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:09:52.204786 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:52.204758    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:09:52.205042 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:52.193908    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-727fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 02 02:09:52.212258 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:52.205178    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm" podUID="a50af6e0-a390-47d4-a4d2-7f90a3055661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 02 02:09:52.216245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:52.212508    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-h8px6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 02 02:09:52.249275 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:52.238295    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6" podUID="ba032348-8064-40c7-8278-0115c943882e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 02 02:09:52.251407 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:52.251335    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr" podStartSLOduration=3.251316805 podStartE2EDuration="3.251316805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:09:52.187827327 +0000 UTC m=+2239.628863824" watchObservedRunningTime="2026-11-02 02:09:52.251316805 +0000 UTC m=+2239.692353295"
Nov 02 02:09:52.743770 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:52.743725    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 02 02:09:52.743995 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:52.743792    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9" podUID="846f35bf-08b7-4447-b7f5-47602b851bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 02 02:09:52.778754 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:52.778716    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 02 02:09:52.779062 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:52.779035    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95" podUID="883a5d43-c155-47d2-bcc0-29a2588b805a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 02 02:09:53.095244 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:53.094620    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ghslr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 02:09:53.095244 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:53.094683    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr" podUID="0c2d802d-8fbd-430d-ae72-29b0e217714d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 02:09:53.188249 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:53.187491    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-727fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 02 02:09:53.188249 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:53.187551    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm" podUID="a50af6e0-a390-47d4-a4d2-7f90a3055661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 02 02:09:53.200307 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:53.196800    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-h8px6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 02 02:09:53.200307 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:53.196863    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6" podUID="ba032348-8064-40c7-8278-0115c943882e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 02 02:09:53.743245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:53.741646    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 02 02:09:53.743245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:53.741708    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9" podUID="846f35bf-08b7-4447-b7f5-47602b851bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 02 02:09:53.751039 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:53.748304    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:09:53.787285 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:53.787250    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 02 02:09:53.787558 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:53.787530    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95" podUID="883a5d43-c155-47d2-bcc0-29a2588b805a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 02 02:09:54.093770 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.093676    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ghslr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 02:09:54.094033 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.094003    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr" podUID="0c2d802d-8fbd-430d-ae72-29b0e217714d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 02:09:54.185888 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.185855    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-727fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 02 02:09:54.186567 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.186523    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm" podUID="a50af6e0-a390-47d4-a4d2-7f90a3055661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 02 02:09:54.208585 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.208501    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-h8px6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 02 02:09:54.208829 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.208800    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6" podUID="ba032348-8064-40c7-8278-0115c943882e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 02 02:09:54.387916 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.387791    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm" podStartSLOduration=5.3877677 podStartE2EDuration="5.3877677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:09:52.362626521 +0000 UTC m=+2239.803663019" watchObservedRunningTime="2026-11-02 02:09:54.3877677 +0000 UTC m=+2241.828804198"
Nov 02 02:09:54.547742 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.547703    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"]
Nov 02 02:09:54.624257 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.624031    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"]
Nov 02 02:09:54.624257 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.624195    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:54.771249 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.755070    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823b5637-9df4-4432-8c27-7de55e83c96e-istiod-ca-cert\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:54.820621 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.820589    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-istio-data\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:54.820915 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.820893    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823b5637-9df4-4432-8c27-7de55e83c96e-istio-podinfo\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:54.821102 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.821087    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-istio-envoy\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:54.821269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.821250    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823b5637-9df4-4432-8c27-7de55e83c96e-istio-token\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:54.821597 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.821465    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-workload-certs\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:54.821758 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.821738    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-workload-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:54.821906 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.821887    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zf26\" (UniqueName: \"kubernetes.io/projected/823b5637-9df4-4432-8c27-7de55e83c96e-kube-api-access-7zf26\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:54.822130 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.822109    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-credential-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:54.822377 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.772335    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:09:54.850562 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.818962    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 02 02:09:54.850562 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.822687    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95" podUID="883a5d43-c155-47d2-bcc0-29a2588b805a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 02 02:09:54.850562 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.820495    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 02 02:09:54.850562 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.822733    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9" podUID="846f35bf-08b7-4447-b7f5-47602b851bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 02 02:09:54.850562 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.772376    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 02:09:54.880337 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.880167    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 02:09:54.940336 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.924860    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-workload-certs\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:54.940336 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.924910    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-workload-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:54.940336 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.924947    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zf26\" (UniqueName: \"kubernetes.io/projected/823b5637-9df4-4432-8c27-7de55e83c96e-kube-api-access-7zf26\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:54.940336 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.925040    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-credential-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:54.940336 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.925086    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823b5637-9df4-4432-8c27-7de55e83c96e-istiod-ca-cert\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:54.940336 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.925125    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-istio-data\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:54.940336 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.925177    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823b5637-9df4-4432-8c27-7de55e83c96e-istio-podinfo\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:54.941643 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.941534    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-workload-certs\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:54.942320 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.942048    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-istio-envoy\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:54.942320 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.942120    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823b5637-9df4-4432-8c27-7de55e83c96e-istio-token\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:54.943247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.943103    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-workload-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:54.944366 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.943765    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-credential-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:54.944521 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.944503    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823b5637-9df4-4432-8c27-7de55e83c96e-istiod-ca-cert\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:54.944845 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.944828    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-istio-data\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:54.996714 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.994511    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-istio-envoy\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:55.021247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:54.998992    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823b5637-9df4-4432-8c27-7de55e83c96e-istio-podinfo\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:55.068292 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:55.065751    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zf26\" (UniqueName: \"kubernetes.io/projected/823b5637-9df4-4432-8c27-7de55e83c96e-kube-api-access-7zf26\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:55.134265 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:55.082463    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823b5637-9df4-4432-8c27-7de55e83c96e-istio-token\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:55.142070 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:55.141260    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ghslr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 02:09:55.142070 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:55.141327    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr" podUID="0c2d802d-8fbd-430d-ae72-29b0e217714d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 02:09:55.198558 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:55.197330    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-h8px6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 02 02:09:55.198558 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:55.197396    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6" podUID="ba032348-8064-40c7-8278-0115c943882e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 02 02:09:55.199246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:55.199204    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-727fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 02 02:09:55.199408 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:55.199385    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm" podUID="a50af6e0-a390-47d4-a4d2-7f90a3055661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 02 02:09:55.228509 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:55.228477    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:55.738666 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:55.738629    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 02 02:09:55.742330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:55.742290    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9" podUID="846f35bf-08b7-4447-b7f5-47602b851bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 02 02:09:55.796402 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:55.796363    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 02 02:09:55.833349 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:55.833300    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95" podUID="883a5d43-c155-47d2-bcc0-29a2588b805a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 02 02:09:55.833672 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:55.833645    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 02:09:55.847638 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:55.834333    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:09:55.874247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:55.848062    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 02:09:55.964247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:55.962324    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64" podUID="36d8fa37-0372-40c4-a7d5-fa490c1c460e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.154:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:09:56.144637 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:56.105728    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ghslr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 02:09:56.144637 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:56.105808    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr" podUID="0c2d802d-8fbd-430d-ae72-29b0e217714d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 02:09:56.230261 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:56.226279    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-h8px6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 02 02:09:56.235477 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:56.235449    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-727fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 02 02:09:56.290281 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:56.290238    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm" podUID="a50af6e0-a390-47d4-a4d2-7f90a3055661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 02 02:09:56.319638 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:56.298408    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6" podUID="ba032348-8064-40c7-8278-0115c943882e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 02 02:09:56.823247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:56.811529    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 02 02:09:56.823247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:56.811614    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95" podUID="883a5d43-c155-47d2-bcc0-29a2588b805a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 02 02:09:56.912248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:56.887581    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:09:56.950344 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:56.949049    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:09:57.021670 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:57.021629    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4"]
Nov 02 02:09:57.059446 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:57.059403    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 02 02:09:57.059746 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:57.059714    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9" podUID="846f35bf-08b7-4447-b7f5-47602b851bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 02 02:09:57.101464 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:09:57.089131    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823b5637_9df4_4432_8c27_7de55e83c96e.slice/crio-1f1f36d18c8c887cd0a251416bd5ebeb3c2c5dc74ae07bb387a397cf938ac1fe WatchSource:0}: Error finding container 1f1f36d18c8c887cd0a251416bd5ebeb3c2c5dc74ae07bb387a397cf938ac1fe: Status 404 returned error can't find the container with id 1f1f36d18c8c887cd0a251416bd5ebeb3c2c5dc74ae07bb387a397cf938ac1fe
Nov 02 02:09:57.221675 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:57.221642    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ghslr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 02:09:57.221913 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:57.221888    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr" podUID="0c2d802d-8fbd-430d-ae72-29b0e217714d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 02:09:57.261252 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:57.250300    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m"]
Nov 02 02:09:57.274248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:57.269566    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:09:57.274248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:57.269672    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:09:57.274248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:57.269721    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:09:57.274248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:57.270369    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-h8px6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 02 02:09:57.274248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:57.270417    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6" podUID="ba032348-8064-40c7-8278-0115c943882e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 02 02:09:57.297290 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:57.278155    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"]
Nov 02 02:09:57.298442 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:57.297540    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-727fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 02 02:09:57.298442 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:57.297608    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm" podUID="a50af6e0-a390-47d4-a4d2-7f90a3055661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 02 02:09:57.595649 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:57.595614    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 02:09:57.726323 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:57.692994    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj" event={"ID":"823b5637-9df4-4432-8c27-7de55e83c96e","Type":"ContainerStarted","Data":"1f1f36d18c8c887cd0a251416bd5ebeb3c2c5dc74ae07bb387a397cf938ac1fe"}
Nov 02 02:09:57.915251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:57.909073    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 02:09:58.119393 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:58.119356    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ghslr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 02:09:58.120035 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:58.119746    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr" podUID="0c2d802d-8fbd-430d-ae72-29b0e217714d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 02:09:58.183861 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:58.183830    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-727fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 02 02:09:58.189416 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:58.189382    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm" podUID="a50af6e0-a390-47d4-a4d2-7f90a3055661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 02 02:09:58.218266 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:58.216377    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-h8px6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 02 02:09:58.218266 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:58.216456    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6" podUID="ba032348-8064-40c7-8278-0115c943882e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 02 02:09:58.796250 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:58.793313    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:09:58.796250 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:58.793378    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95" podUID="883a5d43-c155-47d2-bcc0-29a2588b805a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:09:58.839249 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:58.832830    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:09:58.839249 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:58.832888    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9" podUID="846f35bf-08b7-4447-b7f5-47602b851bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:09:58.862274 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:58.852645    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj" event={"ID":"823b5637-9df4-4432-8c27-7de55e83c96e","Type":"ContainerStarted","Data":"6488d6b44f98f66c5f2f53535711ce77f8d5e4481fabc2568b7676ead16e1365"}
Nov 02 02:09:58.961417 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:58.961373    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64" podUID="36d8fa37-0372-40c4-a7d5-fa490c1c460e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.154:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:09:58.962714 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:58.962618    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:09:59.081919 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:59.081783    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj" podStartSLOduration=5.081756665 podStartE2EDuration="5.081756665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:09:59.028072166 +0000 UTC m=+2246.469108664" watchObservedRunningTime="2026-11-02 02:09:59.081756665 +0000 UTC m=+2246.522793163"
Nov 02 02:09:59.096705 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:59.096658    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ghslr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 02:09:59.096957 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:59.096927    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr" podUID="0c2d802d-8fbd-430d-ae72-29b0e217714d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 02:09:59.122360 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:59.122316    5875 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded"
Nov 02 02:09:59.230563 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:59.230526    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-727fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 02 02:09:59.230891 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:59.230834    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm" podUID="a50af6e0-a390-47d4-a4d2-7f90a3055661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 02 02:09:59.250465 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:59.250418    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-h8px6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 02 02:09:59.250787 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:59.250734    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6" podUID="ba032348-8064-40c7-8278-0115c943882e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 02 02:09:59.251110 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:59.251091    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:09:59.292551 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:59.292496    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 02 02:09:59.292899 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:59.292860    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj" podUID="823b5637-9df4-4432-8c27-7de55e83c96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 02 02:09:59.425905 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:59.424623    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:59.426207 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:59.426106    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:09:59.445156 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:59.445127    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:10:00.003253 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:59.998095    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:10:00.003253 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:09:59.998177    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95" podUID="883a5d43-c155-47d2-bcc0-29a2588b805a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:10:00.117707 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:00.104931    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ghslr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 02:10:00.118174 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:00.118134    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr" podUID="0c2d802d-8fbd-430d-ae72-29b0e217714d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 02:10:00.192464 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:00.192428    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-727fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 02 02:10:00.192797 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:00.192750    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm" podUID="a50af6e0-a390-47d4-a4d2-7f90a3055661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 02 02:10:00.193537 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:00.193511    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-h8px6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 02 02:10:00.193728 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:00.193699    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6" podUID="ba032348-8064-40c7-8278-0115c943882e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 02 02:10:00.266202 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:00.266110    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 02 02:10:00.266942 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:00.266910    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj" podUID="823b5637-9df4-4432-8c27-7de55e83c96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 02 02:10:01.037145 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:01.037102    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:10:01.037792 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:01.037758    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95" podUID="883a5d43-c155-47d2-bcc0-29a2588b805a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:10:01.038620 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:01.038599    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"]
Nov 02 02:10:01.039008 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:01.038985    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95" podUID="883a5d43-c155-47d2-bcc0-29a2588b805a" containerName="istio-proxy" containerID="cri-o://e977dbe1afbe67b9451231612e20f58866314b8d374faa4ad03be04597fe2e5f" gracePeriod=30
Nov 02 02:10:01.101071 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:01.101035    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ghslr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 02:10:01.114782 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:01.114745    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr" podUID="0c2d802d-8fbd-430d-ae72-29b0e217714d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 02:10:01.218562 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:01.218355    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-h8px6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 02 02:10:01.218562 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:01.218416    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6" podUID="ba032348-8064-40c7-8278-0115c943882e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 02 02:10:01.237210 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:01.237097    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 02 02:10:01.237210 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:01.237152    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj" podUID="823b5637-9df4-4432-8c27-7de55e83c96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 02 02:10:01.239793 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:01.239749    5875 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9" secret="" err="secret \"e2e-worker-w5-55bc84-openshift-default-dockercfg-7g4tr\" not found"
Nov 02 02:10:01.242685 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:01.242581    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-727fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 02 02:10:01.242685 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:01.242625    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm" podUID="a50af6e0-a390-47d4-a4d2-7f90a3055661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 02 02:10:01.264214 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:01.256277    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"]
Nov 02 02:10:01.445304 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:01.444563    5875 patch_prober.go:29] interesting pod/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:10:01.445304 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:01.444622    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95" podUID="883a5d43-c155-47d2-bcc0-29a2588b805a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:10:02.046972 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.046923    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4" podUID="4780a97d-8456-4233-83ad-277e4d864125" containerName="payload-processing" containerID="cri-o://5808a9c710ac47f77a05c6145c24fed3be8114f5a6d99c42de65a076e8424b98" gracePeriod=25
Nov 02 02:10:02.093829 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.093790    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ghslr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 02:10:02.093989 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.093856    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr" podUID="0c2d802d-8fbd-430d-ae72-29b0e217714d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 02:10:02.180062 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.180034    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-727fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 02 02:10:02.180204 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.180091    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm" podUID="a50af6e0-a390-47d4-a4d2-7f90a3055661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 02 02:10:02.204250 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.203650    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-h8px6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 02 02:10:02.204250 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.203720    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6" podUID="ba032348-8064-40c7-8278-0115c943882e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 02 02:10:02.249293 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.238616    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 02 02:10:02.249293 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.238677    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj" podUID="823b5637-9df4-4432-8c27-7de55e83c96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 02 02:10:02.259246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.255325    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m" podUID="7b440959-cb4b-4694-b8f2-55a98b1f0e06" containerName="payload-pre-processing" containerID="cri-o://9de8bbc71d6053b0dcc27387e8d3ce450e0fee2218c70e3c8cb40358d1ed1a17" gracePeriod=25
Nov 02 02:10:02.266563 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.266528    5875 generic.go:356] "Generic (PLEG): container finished" podID="4780a97d-8456-4233-83ad-277e4d864125" containerID="5808a9c710ac47f77a05c6145c24fed3be8114f5a6d99c42de65a076e8424b98" exitCode=0
Nov 02 02:10:02.267616 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.266761    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9" podUID="846f35bf-08b7-4447-b7f5-47602b851bd5" containerName="istio-proxy" containerID="cri-o://7d53b1f53020ad0873dc95b099a0c4e80bdddc6db0ebb1a9c0ae8343aafcbe43" gracePeriod=30
Nov 02 02:10:02.267616 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.266895    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4" event={"ID":"4780a97d-8456-4233-83ad-277e4d864125","Type":"ContainerDied","Data":"5808a9c710ac47f77a05c6145c24fed3be8114f5a6d99c42de65a076e8424b98"}
Nov 02 02:10:02.570481 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.570157    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4"
Nov 02 02:10:02.675947 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.675841    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4780a97d-8456-4233-83ad-277e4d864125-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4780a97d-8456-4233-83ad-277e4d864125-plugins-config-volume\") pod \"4780a97d-8456-4233-83ad-277e4d864125\" (UID: \"4780a97d-8456-4233-83ad-277e4d864125\") "
Nov 02 02:10:02.675947 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.675905    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4780a97d-8456-4233-83ad-277e4d864125-kube-api-access-qn7cm\" (UniqueName: \"kubernetes.io/projected/4780a97d-8456-4233-83ad-277e4d864125-kube-api-access-qn7cm\") pod \"4780a97d-8456-4233-83ad-277e4d864125\" (UID: \"4780a97d-8456-4233-83ad-277e4d864125\") "
Nov 02 02:10:02.676642 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.676591    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4780a97d-8456-4233-83ad-277e4d864125-plugins-config-volume" pod "4780a97d-8456-4233-83ad-277e4d864125" (UID: "4780a97d-8456-4233-83ad-277e4d864125"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:10:02.679641 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.679613    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4780a97d-8456-4233-83ad-277e4d864125-kube-api-access-qn7cm" pod "4780a97d-8456-4233-83ad-277e4d864125" (UID: "4780a97d-8456-4233-83ad-277e4d864125"). InnerVolumeSpecName "kube-api-access-qn7cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:10:02.779838 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.779800    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4780a97d-8456-4233-83ad-277e4d864125-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:02.780088 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.780066    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn7cm\" (UniqueName: \"kubernetes.io/projected/4780a97d-8456-4233-83ad-277e4d864125-kube-api-access-qn7cm\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:02.780580 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.780557    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:10:02.908183 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.908156    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m"
Nov 02 02:10:02.982579 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.982544    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b440959-cb4b-4694-b8f2-55a98b1f0e06-kube-api-access-g9rt8\" (UniqueName: \"kubernetes.io/projected/7b440959-cb4b-4694-b8f2-55a98b1f0e06-kube-api-access-g9rt8\") pod \"7b440959-cb4b-4694-b8f2-55a98b1f0e06\" (UID: \"7b440959-cb4b-4694-b8f2-55a98b1f0e06\") "
Nov 02 02:10:02.982794 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.982729    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b440959-cb4b-4694-b8f2-55a98b1f0e06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b440959-cb4b-4694-b8f2-55a98b1f0e06-plugins-config-volume\") pod \"7b440959-cb4b-4694-b8f2-55a98b1f0e06\" (UID: \"7b440959-cb4b-4694-b8f2-55a98b1f0e06\") "
Nov 02 02:10:02.984000 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.983774    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b440959-cb4b-4694-b8f2-55a98b1f0e06-plugins-config-volume" pod "7b440959-cb4b-4694-b8f2-55a98b1f0e06" (UID: "7b440959-cb4b-4694-b8f2-55a98b1f0e06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:10:02.987010 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:02.986983    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b440959-cb4b-4694-b8f2-55a98b1f0e06-kube-api-access-g9rt8" pod "7b440959-cb4b-4694-b8f2-55a98b1f0e06" (UID: "7b440959-cb4b-4694-b8f2-55a98b1f0e06"). InnerVolumeSpecName "kube-api-access-g9rt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:10:03.086262 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:03.084586    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9rt8\" (UniqueName: \"kubernetes.io/projected/7b440959-cb4b-4694-b8f2-55a98b1f0e06-kube-api-access-g9rt8\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:03.086262 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:03.084624    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b440959-cb4b-4694-b8f2-55a98b1f0e06-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:03.237862 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:03.237727    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 02 02:10:03.238127 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:03.238074    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj" podUID="823b5637-9df4-4432-8c27-7de55e83c96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 02 02:10:03.307250 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:03.303493    5875 generic.go:356] "Generic (PLEG): container finished" podID="7b440959-cb4b-4694-b8f2-55a98b1f0e06" containerID="9de8bbc71d6053b0dcc27387e8d3ce450e0fee2218c70e3c8cb40358d1ed1a17" exitCode=0
Nov 02 02:10:03.307250 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:03.303588    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m" event={"ID":"7b440959-cb4b-4694-b8f2-55a98b1f0e06","Type":"ContainerDied","Data":"9de8bbc71d6053b0dcc27387e8d3ce450e0fee2218c70e3c8cb40358d1ed1a17"}
Nov 02 02:10:03.307250 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:03.303636    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m" event={"ID":"7b440959-cb4b-4694-b8f2-55a98b1f0e06","Type":"ContainerDied","Data":"a4474cfa65c5ba863e85abca73d156f9c27d831f6a8f271ab97ba19db600d9ee"}
Nov 02 02:10:03.307250 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:03.303678    5875 scope.go:122] "RemoveContainer" containerID="9de8bbc71d6053b0dcc27387e8d3ce450e0fee2218c70e3c8cb40358d1ed1a17"
Nov 02 02:10:03.307250 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:03.303824    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m"
Nov 02 02:10:03.372816 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:03.372760    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4" event={"ID":"4780a97d-8456-4233-83ad-277e4d864125","Type":"ContainerDied","Data":"c990ddd8510d3df2217088bb108718b3b45bdd31f3f8d378203e9a0cc6e0742c"}
Nov 02 02:10:03.373164 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:03.373120    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4"
Nov 02 02:10:03.464656 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:03.464627    5875 scope.go:122] "RemoveContainer" containerID="9de8bbc71d6053b0dcc27387e8d3ce450e0fee2218c70e3c8cb40358d1ed1a17"
Nov 02 02:10:03.465419 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:10:03.465275    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9de8bbc71d6053b0dcc27387e8d3ce450e0fee2218c70e3c8cb40358d1ed1a17\": container with ID starting with 9de8bbc71d6053b0dcc27387e8d3ce450e0fee2218c70e3c8cb40358d1ed1a17 not found: ID does not exist" containerID="9de8bbc71d6053b0dcc27387e8d3ce450e0fee2218c70e3c8cb40358d1ed1a17"
Nov 02 02:10:03.465419 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:03.465316    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9de8bbc71d6053b0dcc27387e8d3ce450e0fee2218c70e3c8cb40358d1ed1a17"} err="failed to get container status \"9de8bbc71d6053b0dcc27387e8d3ce450e0fee2218c70e3c8cb40358d1ed1a17\": rpc error: code = NotFound desc = could not find container \"9de8bbc71d6053b0dcc27387e8d3ce450e0fee2218c70e3c8cb40358d1ed1a17\": container with ID starting with 9de8bbc71d6053b0dcc27387e8d3ce450e0fee2218c70e3c8cb40358d1ed1a17 not found: ID does not exist"
Nov 02 02:10:03.465419 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:03.465346    5875 scope.go:122] "RemoveContainer" containerID="5808a9c710ac47f77a05c6145c24fed3be8114f5a6d99c42de65a076e8424b98"
Nov 02 02:10:03.575015 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:03.574981    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4"]
Nov 02 02:10:03.582249 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:03.582158    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5700d3d1-574c9d7787-mshl4"]
Nov 02 02:10:03.771824 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:03.770897    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m"]
Nov 02 02:10:03.790340 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:03.786590    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5700d3d1-5f5475c959-pr98m"]
Nov 02 02:10:04.131727 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:04.131633    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:10:04.134083 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:04.134065    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:10:04.138473 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:04.138448    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr"
Nov 02 02:10:04.243250 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:04.242357    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-727fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:10:04.243250 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:04.242435    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm" podUID="a50af6e0-a390-47d4-a4d2-7f90a3055661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:10:04.243580 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:04.243537    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 02 02:10:04.243654 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:04.243586    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj" podUID="823b5637-9df4-4432-8c27-7de55e83c96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 02 02:10:04.273405 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:04.273358    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-h8px6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:10:04.273572 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:04.273416    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6" podUID="ba032348-8064-40c7-8278-0115c943882e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:10:04.374854 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:04.374824    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:10:04.375450 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:04.375344    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:10:04.377190 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:04.377170    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm"
Nov 02 02:10:04.476773 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:04.476747    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:10:04.477156 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:04.477130    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:10:04.478141 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:04.478121    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6"
Nov 02 02:10:04.696992 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:04.696957    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4780a97d-8456-4233-83ad-277e4d864125" path="/var/lib/kubelet/pods/4780a97d-8456-4233-83ad-277e4d864125/volumes"
Nov 02 02:10:04.705248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:04.697567    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b440959-cb4b-4694-b8f2-55a98b1f0e06" path="/var/lib/kubelet/pods/7b440959-cb4b-4694-b8f2-55a98b1f0e06/volumes"
Nov 02 02:10:05.233689 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:05.233650    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 02 02:10:05.234103 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:05.233707    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj" podUID="823b5637-9df4-4432-8c27-7de55e83c96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 02 02:10:06.234373 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.234340    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 02 02:10:06.234771 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.234396    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj" podUID="823b5637-9df4-4432-8c27-7de55e83c96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 02 02:10:06.684900 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.684877    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:10:06.739407 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.739376    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/883a5d43-c155-47d2-bcc0-29a2588b805a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/883a5d43-c155-47d2-bcc0-29a2588b805a-istiod-ca-cert\") pod \"883a5d43-c155-47d2-bcc0-29a2588b805a\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") "
Nov 02 02:10:06.739592 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.739485    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-credential-socket\") pod \"883a5d43-c155-47d2-bcc0-29a2588b805a\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") "
Nov 02 02:10:06.739592 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.739520    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-envoy\") pod \"883a5d43-c155-47d2-bcc0-29a2588b805a\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") "
Nov 02 02:10:06.739712 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.739627    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/883a5d43-c155-47d2-bcc0-29a2588b805a-kube-api-access-gdblr\" (UniqueName: \"kubernetes.io/projected/883a5d43-c155-47d2-bcc0-29a2588b805a-kube-api-access-gdblr\") pod \"883a5d43-c155-47d2-bcc0-29a2588b805a\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") "
Nov 02 02:10:06.739712 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.739679    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-workload-socket\") pod \"883a5d43-c155-47d2-bcc0-29a2588b805a\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") "
Nov 02 02:10:06.739821 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.739750    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-workload-certs\") pod \"883a5d43-c155-47d2-bcc0-29a2588b805a\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") "
Nov 02 02:10:06.739821 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.739774    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-credential-socket" pod "883a5d43-c155-47d2-bcc0-29a2588b805a" (UID: "883a5d43-c155-47d2-bcc0-29a2588b805a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:06.739821 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.739790    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-data\") pod \"883a5d43-c155-47d2-bcc0-29a2588b805a\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") "
Nov 02 02:10:06.739988 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.739858    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-token\" (UniqueName: \"kubernetes.io/projected/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-token\") pod \"883a5d43-c155-47d2-bcc0-29a2588b805a\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") "
Nov 02 02:10:06.739988 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.739896    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-podinfo\") pod \"883a5d43-c155-47d2-bcc0-29a2588b805a\" (UID: \"883a5d43-c155-47d2-bcc0-29a2588b805a\") "
Nov 02 02:10:06.740168 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.739702    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/883a5d43-c155-47d2-bcc0-29a2588b805a-istiod-ca-cert" pod "883a5d43-c155-47d2-bcc0-29a2588b805a" (UID: "883a5d43-c155-47d2-bcc0-29a2588b805a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:10:06.740168 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.739965    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-workload-socket" pod "883a5d43-c155-47d2-bcc0-29a2588b805a" (UID: "883a5d43-c155-47d2-bcc0-29a2588b805a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:06.740168 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.740003    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-workload-certs" pod "883a5d43-c155-47d2-bcc0-29a2588b805a" (UID: "883a5d43-c155-47d2-bcc0-29a2588b805a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:06.740168 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.740049    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-data" pod "883a5d43-c155-47d2-bcc0-29a2588b805a" (UID: "883a5d43-c155-47d2-bcc0-29a2588b805a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:06.740934 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.740903    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:06.740934 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.740928    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:06.741057 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.740942    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:06.741057 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.740956    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/883a5d43-c155-47d2-bcc0-29a2588b805a-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:06.741057 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.740970    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:06.742084 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.742059    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-envoy" pod "883a5d43-c155-47d2-bcc0-29a2588b805a" (UID: "883a5d43-c155-47d2-bcc0-29a2588b805a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:06.742198 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.742141    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-token" pod "883a5d43-c155-47d2-bcc0-29a2588b805a" (UID: "883a5d43-c155-47d2-bcc0-29a2588b805a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:10:06.742494 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.742471    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-podinfo" pod "883a5d43-c155-47d2-bcc0-29a2588b805a" (UID: "883a5d43-c155-47d2-bcc0-29a2588b805a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:10:06.742494 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.742472    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/883a5d43-c155-47d2-bcc0-29a2588b805a-kube-api-access-gdblr" pod "883a5d43-c155-47d2-bcc0-29a2588b805a" (UID: "883a5d43-c155-47d2-bcc0-29a2588b805a"). InnerVolumeSpecName "kube-api-access-gdblr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:10:06.841958 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.841885    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:06.841958 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.841913    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:06.841958 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.841923    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/883a5d43-c155-47d2-bcc0-29a2588b805a-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:06.841958 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:06.841933    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdblr\" (UniqueName: \"kubernetes.io/projected/883a5d43-c155-47d2-bcc0-29a2588b805a-kube-api-access-gdblr\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:07.238918 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.238877    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:10:07.239336 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.239183    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:10:07.239870 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.239852    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:10:07.406196 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.406166    5875 generic.go:356] "Generic (PLEG): container finished" podID="846f35bf-08b7-4447-b7f5-47602b851bd5" containerID="7d53b1f53020ad0873dc95b099a0c4e80bdddc6db0ebb1a9c0ae8343aafcbe43" exitCode=0
Nov 02 02:10:07.406373 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.406253    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9" event={"ID":"846f35bf-08b7-4447-b7f5-47602b851bd5","Type":"ContainerDied","Data":"7d53b1f53020ad0873dc95b099a0c4e80bdddc6db0ebb1a9c0ae8343aafcbe43"}
Nov 02 02:10:07.407853 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.407830    5875 generic.go:356] "Generic (PLEG): container finished" podID="883a5d43-c155-47d2-bcc0-29a2588b805a" containerID="e977dbe1afbe67b9451231612e20f58866314b8d374faa4ad03be04597fe2e5f" exitCode=0
Nov 02 02:10:07.407980 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.407872    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95" event={"ID":"883a5d43-c155-47d2-bcc0-29a2588b805a","Type":"ContainerDied","Data":"e977dbe1afbe67b9451231612e20f58866314b8d374faa4ad03be04597fe2e5f"}
Nov 02 02:10:07.407980 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.407903    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"
Nov 02 02:10:07.407980 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.407923    5875 scope.go:122] "RemoveContainer" containerID="e977dbe1afbe67b9451231612e20f58866314b8d374faa4ad03be04597fe2e5f"
Nov 02 02:10:07.408154 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.407913    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95" event={"ID":"883a5d43-c155-47d2-bcc0-29a2588b805a","Type":"ContainerDied","Data":"ab09cad5bb122042a3b72826e1c65b57a5c2420c0707f8718bebbf302fe145c2"}
Nov 02 02:10:07.417876 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.417856    5875 scope.go:122] "RemoveContainer" containerID="e977dbe1afbe67b9451231612e20f58866314b8d374faa4ad03be04597fe2e5f"
Nov 02 02:10:07.418213 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:10:07.418190    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e977dbe1afbe67b9451231612e20f58866314b8d374faa4ad03be04597fe2e5f\": container with ID starting with e977dbe1afbe67b9451231612e20f58866314b8d374faa4ad03be04597fe2e5f not found: ID does not exist" containerID="e977dbe1afbe67b9451231612e20f58866314b8d374faa4ad03be04597fe2e5f"
Nov 02 02:10:07.418397 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.418282    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e977dbe1afbe67b9451231612e20f58866314b8d374faa4ad03be04597fe2e5f"} err="failed to get container status \"e977dbe1afbe67b9451231612e20f58866314b8d374faa4ad03be04597fe2e5f\": rpc error: code = NotFound desc = could not find container \"e977dbe1afbe67b9451231612e20f58866314b8d374faa4ad03be04597fe2e5f\": container with ID starting with e977dbe1afbe67b9451231612e20f58866314b8d374faa4ad03be04597fe2e5f not found: ID does not exist"
Nov 02 02:10:07.560563 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.560531    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:10:07.597511 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.596564    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"]
Nov 02 02:10:07.603561 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.603535    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-tlc95"]
Nov 02 02:10:07.652185 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.652158    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-workload-socket\") pod \"846f35bf-08b7-4447-b7f5-47602b851bd5\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") "
Nov 02 02:10:07.652344 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.652200    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-credential-socket\") pod \"846f35bf-08b7-4447-b7f5-47602b851bd5\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") "
Nov 02 02:10:07.652344 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.652322    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-envoy\") pod \"846f35bf-08b7-4447-b7f5-47602b851bd5\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") "
Nov 02 02:10:07.652437 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.652405    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/846f35bf-08b7-4447-b7f5-47602b851bd5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/846f35bf-08b7-4447-b7f5-47602b851bd5-istiod-ca-cert\") pod \"846f35bf-08b7-4447-b7f5-47602b851bd5\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") "
Nov 02 02:10:07.652503 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.652487    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-data\") pod \"846f35bf-08b7-4447-b7f5-47602b851bd5\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") "
Nov 02 02:10:07.652556 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.652523    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-workload-socket" pod "846f35bf-08b7-4447-b7f5-47602b851bd5" (UID: "846f35bf-08b7-4447-b7f5-47602b851bd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:07.652613 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.652525    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-podinfo\") pod \"846f35bf-08b7-4447-b7f5-47602b851bd5\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") "
Nov 02 02:10:07.652672 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.652598    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-credential-socket" pod "846f35bf-08b7-4447-b7f5-47602b851bd5" (UID: "846f35bf-08b7-4447-b7f5-47602b851bd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:07.652672 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.652655    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-token\" (UniqueName: \"kubernetes.io/projected/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-token\") pod \"846f35bf-08b7-4447-b7f5-47602b851bd5\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") "
Nov 02 02:10:07.652780 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.652736    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/846f35bf-08b7-4447-b7f5-47602b851bd5-kube-api-access-zqwtg\" (UniqueName: \"kubernetes.io/projected/846f35bf-08b7-4447-b7f5-47602b851bd5-kube-api-access-zqwtg\") pod \"846f35bf-08b7-4447-b7f5-47602b851bd5\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") "
Nov 02 02:10:07.652842 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.652785    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-workload-certs\") pod \"846f35bf-08b7-4447-b7f5-47602b851bd5\" (UID: \"846f35bf-08b7-4447-b7f5-47602b851bd5\") "
Nov 02 02:10:07.652842 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.652799    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-data" pod "846f35bf-08b7-4447-b7f5-47602b851bd5" (UID: "846f35bf-08b7-4447-b7f5-47602b851bd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:07.652842 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.652803    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/846f35bf-08b7-4447-b7f5-47602b851bd5-istiod-ca-cert" pod "846f35bf-08b7-4447-b7f5-47602b851bd5" (UID: "846f35bf-08b7-4447-b7f5-47602b851bd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:10:07.653141 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.653114    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-workload-certs" pod "846f35bf-08b7-4447-b7f5-47602b851bd5" (UID: "846f35bf-08b7-4447-b7f5-47602b851bd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:07.653512 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.653489    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/846f35bf-08b7-4447-b7f5-47602b851bd5-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:07.653597 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.653518    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:07.653597 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.653537    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:07.653597 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.653551    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:07.653597 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.653567    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:07.655044 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.655016    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-podinfo" pod "846f35bf-08b7-4447-b7f5-47602b851bd5" (UID: "846f35bf-08b7-4447-b7f5-47602b851bd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:10:07.655131 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.655050    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-token" pod "846f35bf-08b7-4447-b7f5-47602b851bd5" (UID: "846f35bf-08b7-4447-b7f5-47602b851bd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:10:07.655386 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.655362    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-envoy" pod "846f35bf-08b7-4447-b7f5-47602b851bd5" (UID: "846f35bf-08b7-4447-b7f5-47602b851bd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:07.655386 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.655370    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846f35bf-08b7-4447-b7f5-47602b851bd5-kube-api-access-zqwtg" pod "846f35bf-08b7-4447-b7f5-47602b851bd5" (UID: "846f35bf-08b7-4447-b7f5-47602b851bd5"). InnerVolumeSpecName "kube-api-access-zqwtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:10:07.755122 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.755089    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:07.755122 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.755120    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:07.755122 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.755131    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqwtg\" (UniqueName: \"kubernetes.io/projected/846f35bf-08b7-4447-b7f5-47602b851bd5-kube-api-access-zqwtg\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:07.755122 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:07.755145    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/846f35bf-08b7-4447-b7f5-47602b851bd5-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:08.415300 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:08.415260    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9" event={"ID":"846f35bf-08b7-4447-b7f5-47602b851bd5","Type":"ContainerDied","Data":"f2d76d9a35b5a27acf5324fc155551f412a596fe44707ffa49cd0ada33cc7797"}
Nov 02 02:10:08.415300 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:08.415287    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"
Nov 02 02:10:08.415834 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:08.415370    5875 scope.go:122] "RemoveContainer" containerID="7d53b1f53020ad0873dc95b099a0c4e80bdddc6db0ebb1a9c0ae8343aafcbe43"
Nov 02 02:10:08.580125 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:08.580087    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"]
Nov 02 02:10:08.584306 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:08.584280    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-55bc84-openshift-default-694b575d5c-qwpl9"]
Nov 02 02:10:08.695558 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:08.695475    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="846f35bf-08b7-4447-b7f5-47602b851bd5" path="/var/lib/kubelet/pods/846f35bf-08b7-4447-b7f5-47602b851bd5/volumes"
Nov 02 02:10:08.696056 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:08.696033    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="883a5d43-c155-47d2-bcc0-29a2588b805a" path="/var/lib/kubelet/pods/883a5d43-c155-47d2-bcc0-29a2588b805a/volumes"
Nov 02 02:10:09.506544 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.506510    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"]
Nov 02 02:10:09.507102 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.507088    5875 cpu_manager.go:414] "containerMap: removing container" podUID="7b440959-cb4b-4694-b8f2-55a98b1f0e06" containerName="payload-pre-processing"
Nov 02 02:10:09.507145 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.507103    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b440959-cb4b-4694-b8f2-55a98b1f0e06" containerName="payload-pre-processing"
Nov 02 02:10:09.507145 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.507119    5875 cpu_manager.go:414] "containerMap: removing container" podUID="846f35bf-08b7-4447-b7f5-47602b851bd5" containerName="istio-proxy"
Nov 02 02:10:09.507145 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.507124    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="846f35bf-08b7-4447-b7f5-47602b851bd5" containerName="istio-proxy"
Nov 02 02:10:09.507145 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.507144    5875 cpu_manager.go:414] "containerMap: removing container" podUID="4780a97d-8456-4233-83ad-277e4d864125" containerName="payload-processing"
Nov 02 02:10:09.507333 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.507149    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4780a97d-8456-4233-83ad-277e4d864125" containerName="payload-processing"
Nov 02 02:10:09.507333 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.507158    5875 cpu_manager.go:414] "containerMap: removing container" podUID="883a5d43-c155-47d2-bcc0-29a2588b805a" containerName="istio-proxy"
Nov 02 02:10:09.507333 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.507163    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="883a5d43-c155-47d2-bcc0-29a2588b805a" containerName="istio-proxy"
Nov 02 02:10:09.507333 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.507288    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b440959-cb4b-4694-b8f2-55a98b1f0e06" containerName="payload-pre-processing"
Nov 02 02:10:09.507333 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.507298    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="846f35bf-08b7-4447-b7f5-47602b851bd5" containerName="istio-proxy"
Nov 02 02:10:09.507333 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.507306    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="4780a97d-8456-4233-83ad-277e4d864125" containerName="payload-processing"
Nov 02 02:10:09.507333 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.507312    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="883a5d43-c155-47d2-bcc0-29a2588b805a" containerName="istio-proxy"
Nov 02 02:10:09.512043 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.512019    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.576579 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.576551    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-token\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.576779 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.576598    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-envoy\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.576779 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.576746    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-data\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.576887 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.576823    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh7qf\" (UniqueName: \"kubernetes.io/projected/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-kube-api-access-jh7qf\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.576887 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.576861    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-workload-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.576971 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.576956    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-credential-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.577017 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.576974    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-podinfo\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.577097 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.577080    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istiod-ca-cert\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.577162 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.577137    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-workload-certs\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.678380 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.678349    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-credential-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.678380 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.678382    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-podinfo\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.678624 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.678436    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istiod-ca-cert\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.678624 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.678469    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-workload-certs\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.678624 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.678517    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-token\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.678624 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.678553    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-envoy\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.678856 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.678626    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-data\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.678856 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.678669    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh7qf\" (UniqueName: \"kubernetes.io/projected/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-kube-api-access-jh7qf\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.678856 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.678700    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-workload-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.678856 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.678841    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-workload-certs\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.679088 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.679011    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-credential-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.679507 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.679304    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-data\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.679507 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.679423    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istiod-ca-cert\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.679864 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.679776    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-workload-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.681972 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.681948    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-podinfo\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.682739 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.682715    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-envoy\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.763724 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.763653    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"]
Nov 02 02:10:09.767358 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.767328    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh7qf\" (UniqueName: \"kubernetes.io/projected/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-kube-api-access-jh7qf\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.773401 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.773378    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-token\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:09.834545 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:09.834521    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:10.263527 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:10.263495    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"]
Nov 02 02:10:10.265112 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:10:10.265084    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03078b2_0b08_41d4_9a64_cc50f84bc2a3.slice/crio-f41cd9c0d49deb74777a269d7e727a0191cac7e276156c8101f38a0524e33916 WatchSource:0}: Error finding container f41cd9c0d49deb74777a269d7e727a0191cac7e276156c8101f38a0524e33916: Status 404 returned error can't find the container with id f41cd9c0d49deb74777a269d7e727a0191cac7e276156c8101f38a0524e33916
Nov 02 02:10:10.267685 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:10.267650    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:10:10.267777 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:10.267718    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:10:10.267777 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:10.267746    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:10:10.439368 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:10.438920    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78" event={"ID":"d03078b2-0b08-41d4-9a64-cc50f84bc2a3","Type":"ContainerStarted","Data":"f19c2ef011adce2baa5a730b93da77b0ae47788c2ea6d7f0ed644a23c12de65c"}
Nov 02 02:10:10.439368 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:10.438989    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78" event={"ID":"d03078b2-0b08-41d4-9a64-cc50f84bc2a3","Type":"ContainerStarted","Data":"f41cd9c0d49deb74777a269d7e727a0191cac7e276156c8101f38a0524e33916"}
Nov 02 02:10:10.834659 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:10.834624    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:11.339626 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:10:11.339585    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:10:11.837069 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:11.837039    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 02 02:10:11.837600 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:11.837090    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78" podUID="d03078b2-0b08-41d4-9a64-cc50f84bc2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 02 02:10:12.835796 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:12.835758    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 02 02:10:12.836007 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:12.835819    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78" podUID="d03078b2-0b08-41d4-9a64-cc50f84bc2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 02 02:10:13.834973 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:13.834938    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 02 02:10:13.835437 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:13.834992    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78" podUID="d03078b2-0b08-41d4-9a64-cc50f84bc2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 02 02:10:14.835727 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:14.835691    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 02 02:10:14.836127 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:14.835742    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78" podUID="d03078b2-0b08-41d4-9a64-cc50f84bc2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 02 02:10:15.835971 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:15.835932    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 02 02:10:15.836431 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:15.835988    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78" podUID="d03078b2-0b08-41d4-9a64-cc50f84bc2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 02 02:10:16.719826 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:16.719758    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78" podStartSLOduration=7.719739334 podStartE2EDuration="7.719739334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:10:10.560207331 +0000 UTC m=+2258.001243829" watchObservedRunningTime="2026-11-02 02:10:16.719739334 +0000 UTC m=+2264.160775834"
Nov 02 02:10:16.721205 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:16.721178    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:10:16.835413 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:16.835369    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 02 02:10:16.835603 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:16.835444    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78" podUID="d03078b2-0b08-41d4-9a64-cc50f84bc2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 02 02:10:17.840491 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:17.840454    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:17.840972 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:17.840950    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:17.841724 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:17.841701    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:10:19.260467 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:19.260426    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz"]
Nov 02 02:10:19.260878 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:19.260648    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz" podUID="c1dfbd34-3b3f-477c-994d-6bb41f538eef" containerName="maas-api" containerID="cri-o://84f6b62ee142d4f3ec7ffbbaea9a2fdd6794debe61089678e47f415dad87f681" gracePeriod=30
Nov 02 02:10:19.675457 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:19.631856    5875 generic.go:356] "Generic (PLEG): container finished" podID="c1dfbd34-3b3f-477c-994d-6bb41f538eef" containerID="84f6b62ee142d4f3ec7ffbbaea9a2fdd6794debe61089678e47f415dad87f681" exitCode=0
Nov 02 02:10:19.675457 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:19.632862    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz" event={"ID":"c1dfbd34-3b3f-477c-994d-6bb41f538eef","Type":"ContainerDied","Data":"84f6b62ee142d4f3ec7ffbbaea9a2fdd6794debe61089678e47f415dad87f681"}
Nov 02 02:10:19.863410 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:19.863383    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz"
Nov 02 02:10:19.889642 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:19.889586    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1dfbd34-3b3f-477c-994d-6bb41f538eef-kube-api-access-wbpqv\" (UniqueName: \"kubernetes.io/projected/c1dfbd34-3b3f-477c-994d-6bb41f538eef-kube-api-access-wbpqv\") pod \"c1dfbd34-3b3f-477c-994d-6bb41f538eef\" (UID: \"c1dfbd34-3b3f-477c-994d-6bb41f538eef\") "
Nov 02 02:10:19.889833 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:19.889681    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1dfbd34-3b3f-477c-994d-6bb41f538eef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1dfbd34-3b3f-477c-994d-6bb41f538eef-openshift-service-ca\") pod \"c1dfbd34-3b3f-477c-994d-6bb41f538eef\" (UID: \"c1dfbd34-3b3f-477c-994d-6bb41f538eef\") "
Nov 02 02:10:19.889833 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:19.889721    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1dfbd34-3b3f-477c-994d-6bb41f538eef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1dfbd34-3b3f-477c-994d-6bb41f538eef-maas-api-tls\") pod \"c1dfbd34-3b3f-477c-994d-6bb41f538eef\" (UID: \"c1dfbd34-3b3f-477c-994d-6bb41f538eef\") "
Nov 02 02:10:19.889833 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:19.889799    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1dfbd34-3b3f-477c-994d-6bb41f538eef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1dfbd34-3b3f-477c-994d-6bb41f538eef-metrics-certs\") pod \"c1dfbd34-3b3f-477c-994d-6bb41f538eef\" (UID: \"c1dfbd34-3b3f-477c-994d-6bb41f538eef\") "
Nov 02 02:10:19.890120 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:19.890092    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1dfbd34-3b3f-477c-994d-6bb41f538eef-openshift-service-ca" pod "c1dfbd34-3b3f-477c-994d-6bb41f538eef" (UID: "c1dfbd34-3b3f-477c-994d-6bb41f538eef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:10:19.890492 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:19.890471    5875 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1dfbd34-3b3f-477c-994d-6bb41f538eef-openshift-service-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:19.893645 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:19.893618    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1dfbd34-3b3f-477c-994d-6bb41f538eef-maas-api-tls" pod "c1dfbd34-3b3f-477c-994d-6bb41f538eef" (UID: "c1dfbd34-3b3f-477c-994d-6bb41f538eef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:10:19.894325 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:19.894288    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1dfbd34-3b3f-477c-994d-6bb41f538eef-metrics-certs" pod "c1dfbd34-3b3f-477c-994d-6bb41f538eef" (UID: "c1dfbd34-3b3f-477c-994d-6bb41f538eef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:10:19.894325 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:19.894294    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1dfbd34-3b3f-477c-994d-6bb41f538eef-kube-api-access-wbpqv" pod "c1dfbd34-3b3f-477c-994d-6bb41f538eef" (UID: "c1dfbd34-3b3f-477c-994d-6bb41f538eef"). InnerVolumeSpecName "kube-api-access-wbpqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:10:19.991369 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:19.991332    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1dfbd34-3b3f-477c-994d-6bb41f538eef-maas-api-tls\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:19.991369 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:19.991362    5875 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1dfbd34-3b3f-477c-994d-6bb41f538eef-metrics-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:19.991369 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:19.991376    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbpqv\" (UniqueName: \"kubernetes.io/projected/c1dfbd34-3b3f-477c-994d-6bb41f538eef-kube-api-access-wbpqv\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:20.483716 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:20.483678    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm"]
Nov 02 02:10:20.484137 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:20.483930    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm" podUID="af93eb7b-4ac4-4619-a3b9-958c82ab2280" containerName="maas-api" containerID="cri-o://9ba97d1d50dc4de164f7efc430b90b87ee8bb372b6e75145787adcf3f43b831d" gracePeriod=30
Nov 02 02:10:20.667696 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:20.667659    5875 generic.go:356] "Generic (PLEG): container finished" podID="af93eb7b-4ac4-4619-a3b9-958c82ab2280" containerID="9ba97d1d50dc4de164f7efc430b90b87ee8bb372b6e75145787adcf3f43b831d" exitCode=0
Nov 02 02:10:20.667885 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:20.667788    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm" event={"ID":"af93eb7b-4ac4-4619-a3b9-958c82ab2280","Type":"ContainerDied","Data":"9ba97d1d50dc4de164f7efc430b90b87ee8bb372b6e75145787adcf3f43b831d"}
Nov 02 02:10:20.669277 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:20.669250    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz" event={"ID":"c1dfbd34-3b3f-477c-994d-6bb41f538eef","Type":"ContainerDied","Data":"48b40494f21ce45628dc7d1de596f34432eb104d54f3476ab5b43127742dd39b"}
Nov 02 02:10:20.669277 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:20.669275    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz"
Nov 02 02:10:20.669492 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:20.669301    5875 scope.go:122] "RemoveContainer" containerID="84f6b62ee142d4f3ec7ffbbaea9a2fdd6794debe61089678e47f415dad87f681"
Nov 02 02:10:20.872300 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:20.864280    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz"]
Nov 02 02:10:20.887255 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:20.885192    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-061fab-786f596bbd-xnrmz"]
Nov 02 02:10:21.282512 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:21.282486    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm"
Nov 02 02:10:21.296530 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:21.295840    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af93eb7b-4ac4-4619-a3b9-958c82ab2280-kube-api-access-mv2fs\" (UniqueName: \"kubernetes.io/projected/af93eb7b-4ac4-4619-a3b9-958c82ab2280-kube-api-access-mv2fs\") pod \"af93eb7b-4ac4-4619-a3b9-958c82ab2280\" (UID: \"af93eb7b-4ac4-4619-a3b9-958c82ab2280\") "
Nov 02 02:10:21.296530 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:21.295929    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af93eb7b-4ac4-4619-a3b9-958c82ab2280-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af93eb7b-4ac4-4619-a3b9-958c82ab2280-openshift-service-ca\") pod \"af93eb7b-4ac4-4619-a3b9-958c82ab2280\" (UID: \"af93eb7b-4ac4-4619-a3b9-958c82ab2280\") "
Nov 02 02:10:21.296530 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:21.295958    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af93eb7b-4ac4-4619-a3b9-958c82ab2280-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af93eb7b-4ac4-4619-a3b9-958c82ab2280-maas-api-tls\") pod \"af93eb7b-4ac4-4619-a3b9-958c82ab2280\" (UID: \"af93eb7b-4ac4-4619-a3b9-958c82ab2280\") "
Nov 02 02:10:21.296530 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:21.295989    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af93eb7b-4ac4-4619-a3b9-958c82ab2280-metrics-certs\" (UniqueName: \"kubernetes.io/secret/af93eb7b-4ac4-4619-a3b9-958c82ab2280-metrics-certs\") pod \"af93eb7b-4ac4-4619-a3b9-958c82ab2280\" (UID: \"af93eb7b-4ac4-4619-a3b9-958c82ab2280\") "
Nov 02 02:10:21.296530 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:21.296259    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af93eb7b-4ac4-4619-a3b9-958c82ab2280-openshift-service-ca" pod "af93eb7b-4ac4-4619-a3b9-958c82ab2280" (UID: "af93eb7b-4ac4-4619-a3b9-958c82ab2280"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:10:21.296530 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:21.296499    5875 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af93eb7b-4ac4-4619-a3b9-958c82ab2280-openshift-service-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:21.304255 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:21.301467    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af93eb7b-4ac4-4619-a3b9-958c82ab2280-maas-api-tls" pod "af93eb7b-4ac4-4619-a3b9-958c82ab2280" (UID: "af93eb7b-4ac4-4619-a3b9-958c82ab2280"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:10:21.315800 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:21.315758    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af93eb7b-4ac4-4619-a3b9-958c82ab2280-kube-api-access-mv2fs" pod "af93eb7b-4ac4-4619-a3b9-958c82ab2280" (UID: "af93eb7b-4ac4-4619-a3b9-958c82ab2280"). InnerVolumeSpecName "kube-api-access-mv2fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:10:21.316413 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:21.316356    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af93eb7b-4ac4-4619-a3b9-958c82ab2280-metrics-certs" pod "af93eb7b-4ac4-4619-a3b9-958c82ab2280" (UID: "af93eb7b-4ac4-4619-a3b9-958c82ab2280"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:10:21.397496 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:21.397453    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv2fs\" (UniqueName: \"kubernetes.io/projected/af93eb7b-4ac4-4619-a3b9-958c82ab2280-kube-api-access-mv2fs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:21.403328 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:21.403286    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af93eb7b-4ac4-4619-a3b9-958c82ab2280-maas-api-tls\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:21.403544 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:21.403530    5875 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af93eb7b-4ac4-4619-a3b9-958c82ab2280-metrics-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:21.749882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:21.749833    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm" event={"ID":"af93eb7b-4ac4-4619-a3b9-958c82ab2280","Type":"ContainerDied","Data":"0d61918aef155d31140ba10d0e7ea934ac3098a6ce49b3cc1864d14d48f0a7b9"}
Nov 02 02:10:21.750550 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:21.750532    5875 scope.go:122] "RemoveContainer" containerID="9ba97d1d50dc4de164f7efc430b90b87ee8bb372b6e75145787adcf3f43b831d"
Nov 02 02:10:21.751265 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:21.751244    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm"
Nov 02 02:10:21.930247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:21.927937    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm"]
Nov 02 02:10:21.977462 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:21.977428    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5700d3d1-5788b989d4-ptzgm"]
Nov 02 02:10:22.696895 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:22.696855    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af93eb7b-4ac4-4619-a3b9-958c82ab2280" path="/var/lib/kubelet/pods/af93eb7b-4ac4-4619-a3b9-958c82ab2280/volumes"
Nov 02 02:10:22.717247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:22.709606    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1dfbd34-3b3f-477c-994d-6bb41f538eef" path="/var/lib/kubelet/pods/c1dfbd34-3b3f-477c-994d-6bb41f538eef/volumes"
Nov 02 02:10:24.629027 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.628984    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"]
Nov 02 02:10:24.630013 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.629992    5875 cpu_manager.go:414] "containerMap: removing container" podUID="af93eb7b-4ac4-4619-a3b9-958c82ab2280" containerName="maas-api"
Nov 02 02:10:24.630101 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.630018    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af93eb7b-4ac4-4619-a3b9-958c82ab2280" containerName="maas-api"
Nov 02 02:10:24.630101 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.630059    5875 cpu_manager.go:414] "containerMap: removing container" podUID="c1dfbd34-3b3f-477c-994d-6bb41f538eef" containerName="maas-api"
Nov 02 02:10:24.630101 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.630067    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1dfbd34-3b3f-477c-994d-6bb41f538eef" containerName="maas-api"
Nov 02 02:10:24.630281 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.630198    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="af93eb7b-4ac4-4619-a3b9-958c82ab2280" containerName="maas-api"
Nov 02 02:10:24.630281 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.630216    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1dfbd34-3b3f-477c-994d-6bb41f538eef" containerName="maas-api"
Nov 02 02:10:24.634236 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.634201    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.682050 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.682011    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-token\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.682251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.682087    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-data\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.682315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.682239    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-envoy\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.682315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.682306    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-podinfo\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.682465 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.682440    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istiod-ca-cert\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.682521 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.682501    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-workload-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.682574 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.682532    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j8j6\" (UniqueName: \"kubernetes.io/projected/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-kube-api-access-5j8j6\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.682657 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.682643    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-credential-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.682716 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.682672    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-workload-certs\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.746252 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.744966    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"]
Nov 02 02:10:24.783913 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.783872    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-data\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.784258 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.784205    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-envoy\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.784386 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.784323    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-podinfo\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.784452 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.784436    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istiod-ca-cert\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.784513 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.784497    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-workload-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.784568 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.784525    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j8j6\" (UniqueName: \"kubernetes.io/projected/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-kube-api-access-5j8j6\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.784669 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.784647    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-credential-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.784738 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.784678    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-workload-certs\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.784738 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.784723    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-token\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.784822 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.784789    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-data\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.785613 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.785561    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-credential-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.785731 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.785657    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-workload-socket\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.785809 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.785773    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-workload-certs\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.786518 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.786491    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istiod-ca-cert\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.787884 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.787855    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-envoy\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.788281 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.788249    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-podinfo\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.845862 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.845825    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j8j6\" (UniqueName: \"kubernetes.io/projected/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-kube-api-access-5j8j6\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.846018 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.845913    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-token\") pod \"e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") " pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:24.950913 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:24.950821    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:25.293441 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:25.293394    5875 patch_prober.go:29] interesting pod/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:10:25.293656 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:25.293482    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn" podUID="93d9b083-f014-4998-91b1-a90d3cce473c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:10:25.392836 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:25.392800    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"]
Nov 02 02:10:25.394790 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:10:25.394760    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc55b5b3b_f5f4_4cbf_a7dc_38611a7bf611.slice/crio-9bed96c8f4cec11dc9087390012cf695d9a251e40ed66d1cb1c0e55b9869323d WatchSource:0}: Error finding container 9bed96c8f4cec11dc9087390012cf695d9a251e40ed66d1cb1c0e55b9869323d: Status 404 returned error can't find the container with id 9bed96c8f4cec11dc9087390012cf695d9a251e40ed66d1cb1c0e55b9869323d
Nov 02 02:10:25.397602 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:25.397561    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:10:25.397691 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:25.397672    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:10:25.397741 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:25.397724    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:10:25.786579 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:25.786527    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd" event={"ID":"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611","Type":"ContainerStarted","Data":"b452f792e62a5a483140cd3a1a9a34deff5e564fec59d52ad98f6175f0e9dbab"}
Nov 02 02:10:25.787257 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:25.786588    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd" event={"ID":"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611","Type":"ContainerStarted","Data":"9bed96c8f4cec11dc9087390012cf695d9a251e40ed66d1cb1c0e55b9869323d"}
Nov 02 02:10:25.910369 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:25.910310    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd" podStartSLOduration=1.910295279 podStartE2EDuration="1.910295279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:10:25.908347887 +0000 UTC m=+2273.349384409" watchObservedRunningTime="2026-11-02 02:10:25.910295279 +0000 UTC m=+2273.351331775"
Nov 02 02:10:25.951195 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:25.951098    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:26.953434 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:26.953395    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 02 02:10:26.954019 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:26.953473    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd" podUID="c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 02 02:10:27.825712 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:27.825330    5875 generic.go:356] "Generic (PLEG): container finished" podID="93d9b083-f014-4998-91b1-a90d3cce473c" containerID="28ed2b891f119c8f2542e0c5598765f9ec0988beec097bc0617f6ffceb5c4f5c" exitCode=0
Nov 02 02:10:27.825712 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:27.825379    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn" event={"ID":"93d9b083-f014-4998-91b1-a90d3cce473c","Type":"ContainerDied","Data":"28ed2b891f119c8f2542e0c5598765f9ec0988beec097bc0617f6ffceb5c4f5c"}
Nov 02 02:10:27.826289 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:27.826265    5875 scope.go:122] "RemoveContainer" containerID="28ed2b891f119c8f2542e0c5598765f9ec0988beec097bc0617f6ffceb5c4f5c"
Nov 02 02:10:27.827788 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:27.827754    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:10:27.827892 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:27.827830    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:10:27.827892 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:27.827864    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:10:27.951507 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:27.951471    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 02 02:10:27.951689 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:27.951534    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd" podUID="c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 02 02:10:28.480004 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:28.479945    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"]
Nov 02 02:10:28.835728 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:28.835693    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn" event={"ID":"93d9b083-f014-4998-91b1-a90d3cce473c","Type":"ContainerStarted","Data":"f6cbbed20ef8629aae1323276a23d81d4490b6dd0de6e5442e00795af56c8bd2"}
Nov 02 02:10:28.837564 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:28.837537    5875 generic.go:356] "Generic (PLEG): container finished" podID="d12fc012-f64e-4504-85dc-0bbabf28cb68" containerID="9404db203db3eb9bf9266ed3eb94df6ab4e6a820631fcedd4d864dffeaeb4ff4" exitCode=0
Nov 02 02:10:28.837694 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:28.837604    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt" event={"ID":"d12fc012-f64e-4504-85dc-0bbabf28cb68","Type":"ContainerDied","Data":"9404db203db3eb9bf9266ed3eb94df6ab4e6a820631fcedd4d864dffeaeb4ff4"}
Nov 02 02:10:28.837953 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:28.837940    5875 scope.go:122] "RemoveContainer" containerID="9404db203db3eb9bf9266ed3eb94df6ab4e6a820631fcedd4d864dffeaeb4ff4"
Nov 02 02:10:28.839287 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:28.839261    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:10:28.839363 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:28.839326    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:10:28.839411 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:28.839362    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:10:28.845439 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:28.845419    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:10:28.845687 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:28.845665    5875 patch_prober.go:29] interesting pod/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 02 02:10:28.845731 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:28.845716    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn" podUID="93d9b083-f014-4998-91b1-a90d3cce473c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 02 02:10:28.951212 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:28.951174    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 02 02:10:28.951544 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:28.951514    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd" podUID="c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 02 02:10:29.849248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:29.848837    5875 patch_prober.go:29] interesting pod/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 02 02:10:29.849248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:29.848898    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn" podUID="93d9b083-f014-4998-91b1-a90d3cce473c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 02 02:10:29.883471 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:29.883427    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt" event={"ID":"d12fc012-f64e-4504-85dc-0bbabf28cb68","Type":"ContainerStarted","Data":"60b0da0877514aad2ece5473c0381522077f7dbe303b48dc6dd73a949b6e7ba6"}
Nov 02 02:10:29.962255 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:29.957656    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 02 02:10:29.962255 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:29.957730    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd" podUID="c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 02 02:10:30.727248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:30.718602    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:10:30.727248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:30.720803    5875 patch_prober.go:29] interesting pod/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 02 02:10:30.727248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:30.720857    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt" podUID="d12fc012-f64e-4504-85dc-0bbabf28cb68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 02 02:10:30.850249 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:30.849807    5875 patch_prober.go:29] interesting pod/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 02 02:10:30.850249 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:30.849887    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn" podUID="93d9b083-f014-4998-91b1-a90d3cce473c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 02 02:10:30.956248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:30.952781    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 02 02:10:30.956248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:30.952855    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd" podUID="c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 02 02:10:31.727247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:31.727001    5875 patch_prober.go:29] interesting pod/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 02 02:10:31.727247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:31.727065    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt" podUID="d12fc012-f64e-4504-85dc-0bbabf28cb68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 02 02:10:31.856246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:31.855620    5875 patch_prober.go:29] interesting pod/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 02 02:10:31.856246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:31.855682    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn" podUID="93d9b083-f014-4998-91b1-a90d3cce473c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 02 02:10:32.261248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:32.260568    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:32.279262 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:32.262335    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:32.377688 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:32.371784    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:10:32.790249 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:32.783978    5875 patch_prober.go:29] interesting pod/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 02 02:10:32.790249 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:32.784106    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt" podUID="d12fc012-f64e-4504-85dc-0bbabf28cb68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 02 02:10:33.091934 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:33.091840    5875 patch_prober.go:29] interesting pod/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 02 02:10:33.092472 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:33.092426    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn" podUID="93d9b083-f014-4998-91b1-a90d3cce473c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 02 02:10:33.758083 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:33.758007    5875 patch_prober.go:29] interesting pod/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 02 02:10:33.760133 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:33.760095    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt" podUID="d12fc012-f64e-4504-85dc-0bbabf28cb68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 02 02:10:33.851757 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:33.851707    5875 patch_prober.go:29] interesting pod/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 02 02:10:33.851981 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:33.851793    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn" podUID="93d9b083-f014-4998-91b1-a90d3cce473c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 02 02:10:34.196613 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:34.196528    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:10:34.719248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:34.718706    5875 patch_prober.go:29] interesting pod/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 02 02:10:34.719248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:34.718767    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt" podUID="d12fc012-f64e-4504-85dc-0bbabf28cb68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 02 02:10:34.846131 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:34.846096    5875 patch_prober.go:29] interesting pod/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 02 02:10:34.846350 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:34.846156    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn" podUID="93d9b083-f014-4998-91b1-a90d3cce473c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 02 02:10:35.720760 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:35.720731    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:10:35.721905 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:35.721883    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:10:35.850327 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:35.850293    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:10:35.850583 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:35.850561    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:10:35.851367 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:35.851350    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:10:41.322030 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:10:41.321992    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:10:43.540394 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.540347    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64" podUID="36d8fa37-0372-40c4-a7d5-fa490c1c460e" containerName="main" containerID="cri-o://6a247511470e3283ae59915a530c076388d2c51af41e4275e564ff31cee58798" gracePeriod=45
Nov 02 02:10:43.795940 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.795873    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d426fc-jpcw9"]
Nov 02 02:10:43.800429 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.800407    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d426fc-jpcw9"
Nov 02 02:10:43.801132 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.801112    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:10:43.846332 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.846298    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9dbc65e-d84e-46e7-be0d-6989c268c09f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88d426fc-jpcw9\" (UID: \"e9dbc65e-d84e-46e7-be0d-6989c268c09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d426fc-jpcw9"
Nov 02 02:10:43.846492 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.846417    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg67s\" (UniqueName: \"kubernetes.io/projected/e9dbc65e-d84e-46e7-be0d-6989c268c09f-kube-api-access-hg67s\") pod \"maas-api-revoke-sub-88d426fc-jpcw9\" (UID: \"e9dbc65e-d84e-46e7-be0d-6989c268c09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d426fc-jpcw9"
Nov 02 02:10:43.846548 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.846512    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9dbc65e-d84e-46e7-be0d-6989c268c09f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88d426fc-jpcw9\" (UID: \"e9dbc65e-d84e-46e7-be0d-6989c268c09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d426fc-jpcw9"
Nov 02 02:10:43.859264 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.859202    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d426fc-jpcw9"]
Nov 02 02:10:43.903668 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.903634    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:10:43.903668 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.903646    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:10:43.947709 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.947672    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36d8fa37-0372-40c4-a7d5-fa490c1c460e-kube-api-access-x9vn2\" (UniqueName: \"kubernetes.io/projected/36d8fa37-0372-40c4-a7d5-fa490c1c460e-kube-api-access-x9vn2\") pod \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\" (UID: \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\") "
Nov 02 02:10:43.947873 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.947718    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-home\" (UniqueName: \"kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-home\") pod \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\" (UID: \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\") "
Nov 02 02:10:43.947873 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.947759    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-dshm\") pod \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\" (UID: \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\") "
Nov 02 02:10:43.947873 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.947831    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36d8fa37-0372-40c4-a7d5-fa490c1c460e-tls-certs\" (UniqueName: \"kubernetes.io/secret/36d8fa37-0372-40c4-a7d5-fa490c1c460e-tls-certs\") pod \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\" (UID: \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\") "
Nov 02 02:10:43.948100 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.947880    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-model-cache\") pod \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\" (UID: \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\") "
Nov 02 02:10:43.948100 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.947992    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-tmp-dir\") pod \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\" (UID: \"36d8fa37-0372-40c4-a7d5-fa490c1c460e\") "
Nov 02 02:10:43.948100 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.948051    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-home" pod "36d8fa37-0372-40c4-a7d5-fa490c1c460e" (UID: "36d8fa37-0372-40c4-a7d5-fa490c1c460e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:43.948302 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.948127    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-model-cache" pod "36d8fa37-0372-40c4-a7d5-fa490c1c460e" (UID: "36d8fa37-0372-40c4-a7d5-fa490c1c460e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:43.948361 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.948320    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9dbc65e-d84e-46e7-be0d-6989c268c09f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88d426fc-jpcw9\" (UID: \"e9dbc65e-d84e-46e7-be0d-6989c268c09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d426fc-jpcw9"
Nov 02 02:10:43.948361 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.948333    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-tmp-dir" pod "36d8fa37-0372-40c4-a7d5-fa490c1c460e" (UID: "36d8fa37-0372-40c4-a7d5-fa490c1c460e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:43.948656 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.948559    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9dbc65e-d84e-46e7-be0d-6989c268c09f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88d426fc-jpcw9\" (UID: \"e9dbc65e-d84e-46e7-be0d-6989c268c09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d426fc-jpcw9"
Nov 02 02:10:43.948776 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.948745    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg67s\" (UniqueName: \"kubernetes.io/projected/e9dbc65e-d84e-46e7-be0d-6989c268c09f-kube-api-access-hg67s\") pod \"maas-api-revoke-sub-88d426fc-jpcw9\" (UID: \"e9dbc65e-d84e-46e7-be0d-6989c268c09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d426fc-jpcw9"
Nov 02 02:10:43.948947 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.948916    5875 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-tmp-dir\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:43.948947 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.948935    5875 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-home\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:43.949087 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.948949    5875 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-model-cache\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:43.949282 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.949255    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9dbc65e-d84e-46e7-be0d-6989c268c09f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88d426fc-jpcw9\" (UID: \"e9dbc65e-d84e-46e7-be0d-6989c268c09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d426fc-jpcw9"
Nov 02 02:10:43.950448 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.950423    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-dshm" pod "36d8fa37-0372-40c4-a7d5-fa490c1c460e" (UID: "36d8fa37-0372-40c4-a7d5-fa490c1c460e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:43.950573 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.950478    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36d8fa37-0372-40c4-a7d5-fa490c1c460e-kube-api-access-x9vn2" pod "36d8fa37-0372-40c4-a7d5-fa490c1c460e" (UID: "36d8fa37-0372-40c4-a7d5-fa490c1c460e"). InnerVolumeSpecName "kube-api-access-x9vn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:10:43.950573 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.950515    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36d8fa37-0372-40c4-a7d5-fa490c1c460e-tls-certs" pod "36d8fa37-0372-40c4-a7d5-fa490c1c460e" (UID: "36d8fa37-0372-40c4-a7d5-fa490c1c460e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:10:43.951125 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:43.951103    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9dbc65e-d84e-46e7-be0d-6989c268c09f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88d426fc-jpcw9\" (UID: \"e9dbc65e-d84e-46e7-be0d-6989c268c09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d426fc-jpcw9"
Nov 02 02:10:44.010077 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.010045    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg67s\" (UniqueName: \"kubernetes.io/projected/e9dbc65e-d84e-46e7-be0d-6989c268c09f-kube-api-access-hg67s\") pod \"maas-api-revoke-sub-88d426fc-jpcw9\" (UID: \"e9dbc65e-d84e-46e7-be0d-6989c268c09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d426fc-jpcw9"
Nov 02 02:10:44.050725 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.050639    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9vn2\" (UniqueName: \"kubernetes.io/projected/36d8fa37-0372-40c4-a7d5-fa490c1c460e-kube-api-access-x9vn2\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:44.050725 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.050670    5875 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36d8fa37-0372-40c4-a7d5-fa490c1c460e-dshm\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:44.050725 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.050679    5875 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36d8fa37-0372-40c4-a7d5-fa490c1c460e-tls-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:44.113534 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.113501    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d426fc-jpcw9"
Nov 02 02:10:44.222201 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.222169    5875 generic.go:356] "Generic (PLEG): container finished" podID="36d8fa37-0372-40c4-a7d5-fa490c1c460e" containerID="6a247511470e3283ae59915a530c076388d2c51af41e4275e564ff31cee58798" exitCode=0
Nov 02 02:10:44.222517 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.222215    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"
Nov 02 02:10:44.222517 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.222357    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64" event={"ID":"36d8fa37-0372-40c4-a7d5-fa490c1c460e","Type":"ContainerDied","Data":"6a247511470e3283ae59915a530c076388d2c51af41e4275e564ff31cee58798"}
Nov 02 02:10:44.222517 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.222467    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64" event={"ID":"36d8fa37-0372-40c4-a7d5-fa490c1c460e","Type":"ContainerDied","Data":"5274a062b58bc53320e2a8773917164de7568ec531f907ae33ffc0e04a1013e0"}
Nov 02 02:10:44.222788 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.222556    5875 scope.go:122] "RemoveContainer" containerID="6a247511470e3283ae59915a530c076388d2c51af41e4275e564ff31cee58798"
Nov 02 02:10:44.236637 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.236606    5875 scope.go:122] "RemoveContainer" containerID="6a247511470e3283ae59915a530c076388d2c51af41e4275e564ff31cee58798"
Nov 02 02:10:44.237063 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:10:44.237040    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a247511470e3283ae59915a530c076388d2c51af41e4275e564ff31cee58798\": container with ID starting with 6a247511470e3283ae59915a530c076388d2c51af41e4275e564ff31cee58798 not found: ID does not exist" containerID="6a247511470e3283ae59915a530c076388d2c51af41e4275e564ff31cee58798"
Nov 02 02:10:44.237170 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.237072    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a247511470e3283ae59915a530c076388d2c51af41e4275e564ff31cee58798"} err="failed to get container status \"6a247511470e3283ae59915a530c076388d2c51af41e4275e564ff31cee58798\": rpc error: code = NotFound desc = could not find container \"6a247511470e3283ae59915a530c076388d2c51af41e4275e564ff31cee58798\": container with ID starting with 6a247511470e3283ae59915a530c076388d2c51af41e4275e564ff31cee58798 not found: ID does not exist"
Nov 02 02:10:44.241202 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.241177    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"]
Nov 02 02:10:44.241813 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.241795    5875 cpu_manager.go:414] "containerMap: removing container" podUID="36d8fa37-0372-40c4-a7d5-fa490c1c460e" containerName="main"
Nov 02 02:10:44.241813 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.241812    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d8fa37-0372-40c4-a7d5-fa490c1c460e" containerName="main"
Nov 02 02:10:44.241967 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.241911    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d8fa37-0372-40c4-a7d5-fa490c1c460e" containerName="main"
Nov 02 02:10:44.246562 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.246542    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.300618 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.300523    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-76837e-openshift-default-dockercfg-dpcqs\""
Nov 02 02:10:44.309565 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.308715    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"]
Nov 02 02:10:44.354292 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.354250    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istiod-ca-cert\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.354292 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.354297    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-workload-socket\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.354517 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.354327    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-envoy\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.354517 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.354455    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfkvq\" (UniqueName: \"kubernetes.io/projected/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-kube-api-access-rfkvq\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.354517 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.354499    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-podinfo\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.354677 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.354660    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-workload-certs\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.354786 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.354770    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-credential-socket\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.354868 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.354852    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-data\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.354973 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.354953    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-token\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.457278 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.457043    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-credential-socket\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.457278 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.457157    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-data\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.457278 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.457199    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-token\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.457603 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.457326    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istiod-ca-cert\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.457603 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.457354    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-workload-socket\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.457603 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.457384    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-envoy\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.457603 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.457427    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfkvq\" (UniqueName: \"kubernetes.io/projected/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-kube-api-access-rfkvq\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.457603 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.457452    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-podinfo\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.457603 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.457537    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-workload-certs\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.463817 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.458133    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-workload-certs\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.463817 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.458409    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-credential-socket\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.463817 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.458668    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-data\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.465145 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.464403    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istiod-ca-cert\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.465145 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.464649    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-workload-socket\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.467196 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.467165    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-envoy\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.468790 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.468767    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-podinfo\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.494421 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.494390    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d426fc-jpcw9"]
Nov 02 02:10:44.496348 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:10:44.496323    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9dbc65e_d84e_46e7_be0d_6989c268c09f.slice/crio-e3a1aec2b0bc4ea5c8af18b645b94334b8f85377a9135885025f407757219ddf WatchSource:0}: Error finding container e3a1aec2b0bc4ea5c8af18b645b94334b8f85377a9135885025f407757219ddf: Status 404 returned error can't find the container with id e3a1aec2b0bc4ea5c8af18b645b94334b8f85377a9135885025f407757219ddf
Nov 02 02:10:44.519568 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.519542    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-token\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.525909 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.525888    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfkvq\" (UniqueName: \"kubernetes.io/projected/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-kube-api-access-rfkvq\") pod \"e2e-api-b-76837e-openshift-default-795bd7788c-czsgw\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") " pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.532993 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.532968    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"]
Nov 02 02:10:44.536837 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.536814    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-5700d3d1/e2e-lifecycle-model-5700d3d1-kserve-7b95f4b865-xbs64"]
Nov 02 02:10:44.573428 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.572019    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:44.697601 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:44.697563    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36d8fa37-0372-40c4-a7d5-fa490c1c460e" path="/var/lib/kubelet/pods/36d8fa37-0372-40c4-a7d5-fa490c1c460e/volumes"
Nov 02 02:10:45.019660 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:45.019634    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"]
Nov 02 02:10:45.022593 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:10:45.022568    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21514fb5_5be3_4162_bbfe_7d5a46e4a6f9.slice/crio-d5e3d18b9ebf424c7be3c3fbe88c57a6681a7871a83dce311bb9a982398cb924 WatchSource:0}: Error finding container d5e3d18b9ebf424c7be3c3fbe88c57a6681a7871a83dce311bb9a982398cb924: Status 404 returned error can't find the container with id d5e3d18b9ebf424c7be3c3fbe88c57a6681a7871a83dce311bb9a982398cb924
Nov 02 02:10:45.024967 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:45.024927    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:10:45.025044 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:45.025023    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:10:45.025099 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:45.025066    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:10:45.230698 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:45.230661    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d426fc-jpcw9" event={"ID":"e9dbc65e-d84e-46e7-be0d-6989c268c09f","Type":"ContainerStarted","Data":"2d4a0f53f9f590571cb40ea5526d36121cb41ff2ff53ee5f90408a278a2ba596"}
Nov 02 02:10:45.230913 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:45.230727    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d426fc-jpcw9" event={"ID":"e9dbc65e-d84e-46e7-be0d-6989c268c09f","Type":"ContainerStarted","Data":"e3a1aec2b0bc4ea5c8af18b645b94334b8f85377a9135885025f407757219ddf"}
Nov 02 02:10:45.233386 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:45.233361    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw" event={"ID":"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9","Type":"ContainerStarted","Data":"91ad76a6027fa6435bdfdee6f1de3b16868f36c038e5274351c150a1829f8bbc"}
Nov 02 02:10:45.233386 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:45.233399    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw" event={"ID":"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9","Type":"ContainerStarted","Data":"d5e3d18b9ebf424c7be3c3fbe88c57a6681a7871a83dce311bb9a982398cb924"}
Nov 02 02:10:45.359183 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:45.359091    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d426fc-jpcw9" podStartSLOduration=2.359078582 podStartE2EDuration="2.359078582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:10:45.357304812 +0000 UTC m=+2292.798341310" watchObservedRunningTime="2026-11-02 02:10:45.359078582 +0000 UTC m=+2292.800115078"
Nov 02 02:10:45.481139 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:45.481084    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw" podStartSLOduration=1.481067656 podStartE2EDuration="1.481067656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:10:45.478860631 +0000 UTC m=+2292.919897129" watchObservedRunningTime="2026-11-02 02:10:45.481067656 +0000 UTC m=+2292.922104153"
Nov 02 02:10:45.572714 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:45.572679    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:46.241397 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:46.241356    5875 generic.go:356] "Generic (PLEG): container finished" podID="e9dbc65e-d84e-46e7-be0d-6989c268c09f" containerID="2d4a0f53f9f590571cb40ea5526d36121cb41ff2ff53ee5f90408a278a2ba596" exitCode=0
Nov 02 02:10:46.241796 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:46.241442    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d426fc-jpcw9" event={"ID":"e9dbc65e-d84e-46e7-be0d-6989c268c09f","Type":"ContainerDied","Data":"2d4a0f53f9f590571cb40ea5526d36121cb41ff2ff53ee5f90408a278a2ba596"}
Nov 02 02:10:46.578315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:46.578235    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:47.248022 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:47.247981    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:47.249255 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:47.249236    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:10:47.420647 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:47.420616    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d426fc-jpcw9"
Nov 02 02:10:47.511508 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:47.511413    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9dbc65e-d84e-46e7-be0d-6989c268c09f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9dbc65e-d84e-46e7-be0d-6989c268c09f-maas-api-openshift-ca\") pod \"e9dbc65e-d84e-46e7-be0d-6989c268c09f\" (UID: \"e9dbc65e-d84e-46e7-be0d-6989c268c09f\") "
Nov 02 02:10:47.511695 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:47.511578    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9dbc65e-d84e-46e7-be0d-6989c268c09f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9dbc65e-d84e-46e7-be0d-6989c268c09f-maas-api-xks-ca\") pod \"e9dbc65e-d84e-46e7-be0d-6989c268c09f\" (UID: \"e9dbc65e-d84e-46e7-be0d-6989c268c09f\") "
Nov 02 02:10:47.511764 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:47.511738    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9dbc65e-d84e-46e7-be0d-6989c268c09f-kube-api-access-hg67s\" (UniqueName: \"kubernetes.io/projected/e9dbc65e-d84e-46e7-be0d-6989c268c09f-kube-api-access-hg67s\") pod \"e9dbc65e-d84e-46e7-be0d-6989c268c09f\" (UID: \"e9dbc65e-d84e-46e7-be0d-6989c268c09f\") "
Nov 02 02:10:47.511764 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:47.511749    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9dbc65e-d84e-46e7-be0d-6989c268c09f-maas-api-openshift-ca" pod "e9dbc65e-d84e-46e7-be0d-6989c268c09f" (UID: "e9dbc65e-d84e-46e7-be0d-6989c268c09f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:10:47.512311 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:47.512214    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9dbc65e-d84e-46e7-be0d-6989c268c09f-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:47.514021 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:47.513996    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9dbc65e-d84e-46e7-be0d-6989c268c09f-maas-api-xks-ca" pod "e9dbc65e-d84e-46e7-be0d-6989c268c09f" (UID: "e9dbc65e-d84e-46e7-be0d-6989c268c09f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:10:47.514021 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:47.514004    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9dbc65e-d84e-46e7-be0d-6989c268c09f-kube-api-access-hg67s" pod "e9dbc65e-d84e-46e7-be0d-6989c268c09f" (UID: "e9dbc65e-d84e-46e7-be0d-6989c268c09f"). InnerVolumeSpecName "kube-api-access-hg67s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:10:47.613297 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:47.613253    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg67s\" (UniqueName: \"kubernetes.io/projected/e9dbc65e-d84e-46e7-be0d-6989c268c09f-kube-api-access-hg67s\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:47.613297 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:47.613290    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9dbc65e-d84e-46e7-be0d-6989c268c09f-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:48.255758 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:48.255720    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d426fc-jpcw9"
Nov 02 02:10:48.255758 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:48.255751    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d426fc-jpcw9" event={"ID":"e9dbc65e-d84e-46e7-be0d-6989c268c09f","Type":"ContainerDied","Data":"e3a1aec2b0bc4ea5c8af18b645b94334b8f85377a9135885025f407757219ddf"}
Nov 02 02:10:48.256211 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:48.255799    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3a1aec2b0bc4ea5c8af18b645b94334b8f85377a9135885025f407757219ddf"
Nov 02 02:10:54.452468 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:54.452431    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp"]
Nov 02 02:10:54.455189 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:54.455161    5875 cpu_manager.go:414] "containerMap: removing container" podUID="e9dbc65e-d84e-46e7-be0d-6989c268c09f" containerName="revoke-keys"
Nov 02 02:10:54.455376 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:54.455361    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9dbc65e-d84e-46e7-be0d-6989c268c09f" containerName="revoke-keys"
Nov 02 02:10:54.455706 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:54.455693    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9dbc65e-d84e-46e7-be0d-6989c268c09f" containerName="revoke-keys"
Nov 02 02:10:54.465513 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:54.465488    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp"
Nov 02 02:10:54.481692 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:54.481664    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp"]
Nov 02 02:10:54.523947 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:54.520575    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-76837e\""
Nov 02 02:10:54.523947 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:54.520644    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-76837e-dockercfg-tgt6t\""
Nov 02 02:10:54.534457 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:54.533352    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:10:54.572035 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:54.572000    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8"]
Nov 02 02:10:54.579617 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:54.579590    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8"
Nov 02 02:10:54.628107 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:54.628063    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72m28\" (UniqueName: \"kubernetes.io/projected/2584d15c-d3a3-4392-bb37-5b2c650b7cf7-kube-api-access-72m28\") pod \"payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp\" (UID: \"2584d15c-d3a3-4392-bb37-5b2c650b7cf7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp"
Nov 02 02:10:54.628107 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:54.628110    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2584d15c-d3a3-4392-bb37-5b2c650b7cf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp\" (UID: \"2584d15c-d3a3-4392-bb37-5b2c650b7cf7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp"
Nov 02 02:10:54.635283 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:54.635253    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8"]
Nov 02 02:10:54.730112 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:54.730068    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11cd3f02-9989-4e40-be11-f7e7b72f9275-plugins-config-volume\") pod \"payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8\" (UID: \"11cd3f02-9989-4e40-be11-f7e7b72f9275\") " pod="openshift-ingress/payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8"
Nov 02 02:10:54.730552 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:54.730528    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx52x\" (UniqueName: \"kubernetes.io/projected/11cd3f02-9989-4e40-be11-f7e7b72f9275-kube-api-access-tx52x\") pod \"payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8\" (UID: \"11cd3f02-9989-4e40-be11-f7e7b72f9275\") " pod="openshift-ingress/payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8"
Nov 02 02:10:54.734251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:54.730691    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72m28\" (UniqueName: \"kubernetes.io/projected/2584d15c-d3a3-4392-bb37-5b2c650b7cf7-kube-api-access-72m28\") pod \"payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp\" (UID: \"2584d15c-d3a3-4392-bb37-5b2c650b7cf7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp"
Nov 02 02:10:54.734420 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:54.734399    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2584d15c-d3a3-4392-bb37-5b2c650b7cf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp\" (UID: \"2584d15c-d3a3-4392-bb37-5b2c650b7cf7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp"
Nov 02 02:10:54.742025 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:54.741998    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2584d15c-d3a3-4392-bb37-5b2c650b7cf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp\" (UID: \"2584d15c-d3a3-4392-bb37-5b2c650b7cf7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp"
Nov 02 02:10:54.803212 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:54.799538    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72m28\" (UniqueName: \"kubernetes.io/projected/2584d15c-d3a3-4392-bb37-5b2c650b7cf7-kube-api-access-72m28\") pod \"payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp\" (UID: \"2584d15c-d3a3-4392-bb37-5b2c650b7cf7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp"
Nov 02 02:10:54.816202 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:54.816169    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp"
Nov 02 02:10:54.836156 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:54.836128    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx52x\" (UniqueName: \"kubernetes.io/projected/11cd3f02-9989-4e40-be11-f7e7b72f9275-kube-api-access-tx52x\") pod \"payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8\" (UID: \"11cd3f02-9989-4e40-be11-f7e7b72f9275\") " pod="openshift-ingress/payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8"
Nov 02 02:10:54.836360 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:54.836336    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11cd3f02-9989-4e40-be11-f7e7b72f9275-plugins-config-volume\") pod \"payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8\" (UID: \"11cd3f02-9989-4e40-be11-f7e7b72f9275\") " pod="openshift-ingress/payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8"
Nov 02 02:10:54.837249 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:54.837202    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11cd3f02-9989-4e40-be11-f7e7b72f9275-plugins-config-volume\") pod \"payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8\" (UID: \"11cd3f02-9989-4e40-be11-f7e7b72f9275\") " pod="openshift-ingress/payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8"
Nov 02 02:10:54.898655 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:54.898624    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx52x\" (UniqueName: \"kubernetes.io/projected/11cd3f02-9989-4e40-be11-f7e7b72f9275-kube-api-access-tx52x\") pod \"payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8\" (UID: \"11cd3f02-9989-4e40-be11-f7e7b72f9275\") " pod="openshift-ingress/payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8"
Nov 02 02:10:55.193734 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:55.193702    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp"]
Nov 02 02:10:55.195309 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:10:55.195281    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2584d15c_d3a3_4392_bb37_5b2c650b7cf7.slice/crio-d5007fc033c2df252be089b252c4ff9449f2138c9aaa11b98858a28051d8e22b WatchSource:0}: Error finding container d5007fc033c2df252be089b252c4ff9449f2138c9aaa11b98858a28051d8e22b: Status 404 returned error can't find the container with id d5007fc033c2df252be089b252c4ff9449f2138c9aaa11b98858a28051d8e22b
Nov 02 02:10:55.197165 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:55.197144    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8"
Nov 02 02:10:55.329043 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:55.328640    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp" event={"ID":"2584d15c-d3a3-4392-bb37-5b2c650b7cf7","Type":"ContainerStarted","Data":"06c7a2f5ca4e3fdd3f6ead4ee54547d54881443ca0c85d542df36f68e77d867d"}
Nov 02 02:10:55.329043 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:55.328713    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp"
Nov 02 02:10:55.329043 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:55.328740    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp" event={"ID":"2584d15c-d3a3-4392-bb37-5b2c650b7cf7","Type":"ContainerStarted","Data":"d5007fc033c2df252be089b252c4ff9449f2138c9aaa11b98858a28051d8e22b"}
Nov 02 02:10:55.448385 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:55.448265    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp" podStartSLOduration=1.448211965 podStartE2EDuration="1.448211965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:10:55.447264386 +0000 UTC m=+2302.888300884" watchObservedRunningTime="2026-11-02 02:10:55.448211965 +0000 UTC m=+2302.889248463"
Nov 02 02:10:55.600418 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:55.600385    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8"]
Nov 02 02:10:55.604707 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:10:55.604677    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11cd3f02_9989_4e40_be11_f7e7b72f9275.slice/crio-22ea978c6340f00ec05585315c68c4c9fad552202f5e626019ef3251af223b45 WatchSource:0}: Error finding container 22ea978c6340f00ec05585315c68c4c9fad552202f5e626019ef3251af223b45: Status 404 returned error can't find the container with id 22ea978c6340f00ec05585315c68c4c9fad552202f5e626019ef3251af223b45
Nov 02 02:10:56.340147 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:56.340085    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8" event={"ID":"11cd3f02-9989-4e40-be11-f7e7b72f9275","Type":"ContainerStarted","Data":"d9b64da82181e4aa17563fd09b03c46a6a2788c06954e0ae8955eb15e866caee"}
Nov 02 02:10:56.340396 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:56.340203    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8" event={"ID":"11cd3f02-9989-4e40-be11-f7e7b72f9275","Type":"ContainerStarted","Data":"22ea978c6340f00ec05585315c68c4c9fad552202f5e626019ef3251af223b45"}
Nov 02 02:10:56.340499 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:56.340485    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8"
Nov 02 02:10:56.456704 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:10:56.456637    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8" podStartSLOduration=2.456615512 podStartE2EDuration="2.456615512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:10:56.455637374 +0000 UTC m=+2303.896673890" watchObservedRunningTime="2026-11-02 02:10:56.456615512 +0000 UTC m=+2303.897652014"
Nov 02 02:11:06.343934 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:06.343907    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp"
Nov 02 02:11:07.352537 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:07.352503    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8"
Nov 02 02:11:11.320477 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:11:11.320437    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:11:25.540873 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.540836    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"]
Nov 02 02:11:25.544914 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.544893    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:11:25.649144 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.649083    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4211c0\"/\"kube-root-ca.crt\""
Nov 02 02:11:25.649144 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.649129    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4211c0\"/\"test-model-4211c0-kserve-self-signed-certs\""
Nov 02 02:11:25.649554 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.649528    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4211c0\"/\"default-dockercfg-jmjzv\""
Nov 02 02:11:25.649699 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.649573    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4211c0\"/\"openshift-service-ca.crt\""
Nov 02 02:11:25.651904 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.651880    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r26tc\" (UniqueName: \"kubernetes.io/projected/c30b1225-cfec-4f91-9fa0-ab84afcb0598-kube-api-access-r26tc\") pod \"test-model-4211c0-kserve-687b7f8985-5x9z4\" (UID: \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\") " pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:11:25.652022 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.651920    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-home\") pod \"test-model-4211c0-kserve-687b7f8985-5x9z4\" (UID: \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\") " pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:11:25.652022 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.651998    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-model-cache\") pod \"test-model-4211c0-kserve-687b7f8985-5x9z4\" (UID: \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\") " pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:11:25.652169 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.652152    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-dshm\") pod \"test-model-4211c0-kserve-687b7f8985-5x9z4\" (UID: \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\") " pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:11:25.652273 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.652183    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-tmp-dir\") pod \"test-model-4211c0-kserve-687b7f8985-5x9z4\" (UID: \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\") " pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:11:25.652273 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.652260    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"]
Nov 02 02:11:25.652396 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.652322    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30b1225-cfec-4f91-9fa0-ab84afcb0598-tls-certs\") pod \"test-model-4211c0-kserve-687b7f8985-5x9z4\" (UID: \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\") " pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:11:25.753572 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.753532    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30b1225-cfec-4f91-9fa0-ab84afcb0598-tls-certs\") pod \"test-model-4211c0-kserve-687b7f8985-5x9z4\" (UID: \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\") " pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:11:25.753791 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.753601    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r26tc\" (UniqueName: \"kubernetes.io/projected/c30b1225-cfec-4f91-9fa0-ab84afcb0598-kube-api-access-r26tc\") pod \"test-model-4211c0-kserve-687b7f8985-5x9z4\" (UID: \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\") " pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:11:25.753791 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.753645    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-home\") pod \"test-model-4211c0-kserve-687b7f8985-5x9z4\" (UID: \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\") " pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:11:25.753791 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.753689    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-model-cache\") pod \"test-model-4211c0-kserve-687b7f8985-5x9z4\" (UID: \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\") " pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:11:25.753986 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.753910    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-dshm\") pod \"test-model-4211c0-kserve-687b7f8985-5x9z4\" (UID: \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\") " pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:11:25.753986 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.753954    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-tmp-dir\") pod \"test-model-4211c0-kserve-687b7f8985-5x9z4\" (UID: \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\") " pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:11:25.754171 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.754146    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-home\") pod \"test-model-4211c0-kserve-687b7f8985-5x9z4\" (UID: \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\") " pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:11:25.754254 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.754212    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-model-cache\") pod \"test-model-4211c0-kserve-687b7f8985-5x9z4\" (UID: \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\") " pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:11:25.754450 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.754419    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-tmp-dir\") pod \"test-model-4211c0-kserve-687b7f8985-5x9z4\" (UID: \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\") " pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:11:25.756765 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.756744    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-dshm\") pod \"test-model-4211c0-kserve-687b7f8985-5x9z4\" (UID: \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\") " pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:11:25.756864 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.756762    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30b1225-cfec-4f91-9fa0-ab84afcb0598-tls-certs\") pod \"test-model-4211c0-kserve-687b7f8985-5x9z4\" (UID: \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\") " pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:11:25.814829 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.814757    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r26tc\" (UniqueName: \"kubernetes.io/projected/c30b1225-cfec-4f91-9fa0-ab84afcb0598-kube-api-access-r26tc\") pod \"test-model-4211c0-kserve-687b7f8985-5x9z4\" (UID: \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\") " pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:11:25.858618 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:25.858583    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:11:26.295683 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:26.295655    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"]
Nov 02 02:11:26.296684 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:11:26.296656    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc30b1225_cfec_4f91_9fa0_ab84afcb0598.slice/crio-9a40ebcccd734e902d31e6ea7fc61a5682f205d0312e3e790b4485042be85359 WatchSource:0}: Error finding container 9a40ebcccd734e902d31e6ea7fc61a5682f205d0312e3e790b4485042be85359: Status 404 returned error can't find the container with id 9a40ebcccd734e902d31e6ea7fc61a5682f205d0312e3e790b4485042be85359
Nov 02 02:11:26.565128 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:26.565035    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4" event={"ID":"c30b1225-cfec-4f91-9fa0-ab84afcb0598","Type":"ContainerStarted","Data":"fda5f7baed2ffdfd1a83105a2a3e29cc14d7c09299ee959bfdd8bccb1b482706"}
Nov 02 02:11:26.565128 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:26.565082    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4" event={"ID":"c30b1225-cfec-4f91-9fa0-ab84afcb0598","Type":"ContainerStarted","Data":"9a40ebcccd734e902d31e6ea7fc61a5682f205d0312e3e790b4485042be85359"}
Nov 02 02:11:26.681351 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:26.681298    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4" podStartSLOduration=1.6812830029999999 podStartE2EDuration="1.681283003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:11:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:11:26.680048947 +0000 UTC m=+2334.121085445" watchObservedRunningTime="2026-11-02 02:11:26.681283003 +0000 UTC m=+2334.122319503"
Nov 02 02:11:26.859320 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:26.859211    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:11:27.125339 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:27.125183    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"]
Nov 02 02:11:27.125521 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:27.125495    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj" podUID="823b5637-9df4-4432-8c27-7de55e83c96e" containerName="istio-proxy" containerID="cri-o://6488d6b44f98f66c5f2f53535711ce77f8d5e4481fabc2568b7676ead16e1365" gracePeriod=30
Nov 02 02:11:27.136843 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:27.136815    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"]
Nov 02 02:11:27.137014 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:27.136993    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd" podUID="c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611" containerName="istio-proxy" containerID="cri-o://b452f792e62a5a483140cd3a1a9a34deff5e564fec59d52ad98f6175f0e9dbab" gracePeriod=30
Nov 02 02:11:27.149126 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:27.149092    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"]
Nov 02 02:11:27.149327 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:27.149306    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs" podUID="c764b207-bd1e-4652-b4c1-519c8f453aa2" containerName="istio-proxy" containerID="cri-o://0b62cee84c26747b0a06eaec209471c1f8029cb967b1598169949f1ef40ea83f" gracePeriod=30
Nov 02 02:11:27.174719 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:27.174680    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"]
Nov 02 02:11:27.174907 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:27.174885    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78" podUID="d03078b2-0b08-41d4-9a64-cc50f84bc2a3" containerName="istio-proxy" containerID="cri-o://f19c2ef011adce2baa5a730b93da77b0ae47788c2ea6d7f0ed644a23c12de65c" gracePeriod=30
Nov 02 02:11:31.443163 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:31.443126    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:11:32.263336 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.263309    5875 patch_prober.go:29] interesting pod/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 02 02:11:32.263455 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.263373    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd" podUID="c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 02 02:11:32.485584 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.485560    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:11:32.518913 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.518874    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 02:11:32.548283 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.548256    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-workload-certs\") pod \"823b5637-9df4-4432-8c27-7de55e83c96e\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") "
Nov 02 02:11:32.548449 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.548333    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-istio-envoy\") pod \"823b5637-9df4-4432-8c27-7de55e83c96e\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") "
Nov 02 02:11:32.548449 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.548372    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/823b5637-9df4-4432-8c27-7de55e83c96e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823b5637-9df4-4432-8c27-7de55e83c96e-istio-podinfo\") pod \"823b5637-9df4-4432-8c27-7de55e83c96e\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") "
Nov 02 02:11:32.548449 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.548402    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-workload-socket\") pod \"823b5637-9df4-4432-8c27-7de55e83c96e\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") "
Nov 02 02:11:32.549150 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.548491    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-istio-data\") pod \"823b5637-9df4-4432-8c27-7de55e83c96e\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") "
Nov 02 02:11:32.549150 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.548530    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-credential-socket\") pod \"823b5637-9df4-4432-8c27-7de55e83c96e\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") "
Nov 02 02:11:32.549150 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.548632    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/823b5637-9df4-4432-8c27-7de55e83c96e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823b5637-9df4-4432-8c27-7de55e83c96e-istiod-ca-cert\") pod \"823b5637-9df4-4432-8c27-7de55e83c96e\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") "
Nov 02 02:11:32.549150 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.548673    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823b5637-9df4-4432-8c27-7de55e83c96e-istio-token\" (UniqueName: \"kubernetes.io/projected/823b5637-9df4-4432-8c27-7de55e83c96e-istio-token\") pod \"823b5637-9df4-4432-8c27-7de55e83c96e\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") "
Nov 02 02:11:32.549150 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.548775    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823b5637-9df4-4432-8c27-7de55e83c96e-kube-api-access-7zf26\" (UniqueName: \"kubernetes.io/projected/823b5637-9df4-4432-8c27-7de55e83c96e-kube-api-access-7zf26\") pod \"823b5637-9df4-4432-8c27-7de55e83c96e\" (UID: \"823b5637-9df4-4432-8c27-7de55e83c96e\") "
Nov 02 02:11:32.549150 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.548912    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-workload-certs" pod "823b5637-9df4-4432-8c27-7de55e83c96e" (UID: "823b5637-9df4-4432-8c27-7de55e83c96e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.550197 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.549406    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.550197 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.549427    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-istio-data" pod "823b5637-9df4-4432-8c27-7de55e83c96e" (UID: "823b5637-9df4-4432-8c27-7de55e83c96e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.550197 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.549482    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-workload-socket" pod "823b5637-9df4-4432-8c27-7de55e83c96e" (UID: "823b5637-9df4-4432-8c27-7de55e83c96e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.550197 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.549638    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-credential-socket" pod "823b5637-9df4-4432-8c27-7de55e83c96e" (UID: "823b5637-9df4-4432-8c27-7de55e83c96e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.550197 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.549814    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823b5637-9df4-4432-8c27-7de55e83c96e-istiod-ca-cert" pod "823b5637-9df4-4432-8c27-7de55e83c96e" (UID: "823b5637-9df4-4432-8c27-7de55e83c96e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:11:32.551557 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.551526    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823b5637-9df4-4432-8c27-7de55e83c96e-kube-api-access-7zf26" pod "823b5637-9df4-4432-8c27-7de55e83c96e" (UID: "823b5637-9df4-4432-8c27-7de55e83c96e"). InnerVolumeSpecName "kube-api-access-7zf26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:32.551670 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.551603    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/823b5637-9df4-4432-8c27-7de55e83c96e-istio-podinfo" pod "823b5637-9df4-4432-8c27-7de55e83c96e" (UID: "823b5637-9df4-4432-8c27-7de55e83c96e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:11:32.553692 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.553588    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-istio-envoy" pod "823b5637-9df4-4432-8c27-7de55e83c96e" (UID: "823b5637-9df4-4432-8c27-7de55e83c96e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.554559 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.554507    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823b5637-9df4-4432-8c27-7de55e83c96e-istio-token" pod "823b5637-9df4-4432-8c27-7de55e83c96e" (UID: "823b5637-9df4-4432-8c27-7de55e83c96e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:32.555000 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.554977    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:11:32.576359 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.575621    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:11:32.607432 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.607394    5875 generic.go:356] "Generic (PLEG): container finished" podID="823b5637-9df4-4432-8c27-7de55e83c96e" containerID="6488d6b44f98f66c5f2f53535711ce77f8d5e4481fabc2568b7676ead16e1365" exitCode=0
Nov 02 02:11:32.607624 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.607453    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"
Nov 02 02:11:32.607624 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.607476    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj" event={"ID":"823b5637-9df4-4432-8c27-7de55e83c96e","Type":"ContainerDied","Data":"6488d6b44f98f66c5f2f53535711ce77f8d5e4481fabc2568b7676ead16e1365"}
Nov 02 02:11:32.607624 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.607530    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj" event={"ID":"823b5637-9df4-4432-8c27-7de55e83c96e","Type":"ContainerDied","Data":"1f1f36d18c8c887cd0a251416bd5ebeb3c2c5dc74ae07bb387a397cf938ac1fe"}
Nov 02 02:11:32.607624 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.607546    5875 scope.go:122] "RemoveContainer" containerID="6488d6b44f98f66c5f2f53535711ce77f8d5e4481fabc2568b7676ead16e1365"
Nov 02 02:11:32.608805 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.608778    5875 generic.go:356] "Generic (PLEG): container finished" podID="c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611" containerID="b452f792e62a5a483140cd3a1a9a34deff5e564fec59d52ad98f6175f0e9dbab" exitCode=0
Nov 02 02:11:32.608923 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.608807    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd" event={"ID":"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611","Type":"ContainerDied","Data":"b452f792e62a5a483140cd3a1a9a34deff5e564fec59d52ad98f6175f0e9dbab"}
Nov 02 02:11:32.608923 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.608834    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"
Nov 02 02:11:32.608923 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.608857    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd" event={"ID":"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611","Type":"ContainerDied","Data":"9bed96c8f4cec11dc9087390012cf695d9a251e40ed66d1cb1c0e55b9869323d"}
Nov 02 02:11:32.610506 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.610483    5875 generic.go:356] "Generic (PLEG): container finished" podID="d03078b2-0b08-41d4-9a64-cc50f84bc2a3" containerID="f19c2ef011adce2baa5a730b93da77b0ae47788c2ea6d7f0ed644a23c12de65c" exitCode=0
Nov 02 02:11:32.610632 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.610512    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78" event={"ID":"d03078b2-0b08-41d4-9a64-cc50f84bc2a3","Type":"ContainerDied","Data":"f19c2ef011adce2baa5a730b93da77b0ae47788c2ea6d7f0ed644a23c12de65c"}
Nov 02 02:11:32.610632 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.610560    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78" event={"ID":"d03078b2-0b08-41d4-9a64-cc50f84bc2a3","Type":"ContainerDied","Data":"f41cd9c0d49deb74777a269d7e727a0191cac7e276156c8101f38a0524e33916"}
Nov 02 02:11:32.610632 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.610520    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"
Nov 02 02:11:32.612490 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.612464    5875 generic.go:356] "Generic (PLEG): container finished" podID="c764b207-bd1e-4652-b4c1-519c8f453aa2" containerID="0b62cee84c26747b0a06eaec209471c1f8029cb967b1598169949f1ef40ea83f" exitCode=0
Nov 02 02:11:32.612614 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.612511    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"
Nov 02 02:11:32.612614 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.612550    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs" event={"ID":"c764b207-bd1e-4652-b4c1-519c8f453aa2","Type":"ContainerDied","Data":"0b62cee84c26747b0a06eaec209471c1f8029cb967b1598169949f1ef40ea83f"}
Nov 02 02:11:32.612614 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.612598    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs" event={"ID":"c764b207-bd1e-4652-b4c1-519c8f453aa2","Type":"ContainerDied","Data":"72775ef79e14cbb905b79693bb3740dd134ab4fc395e1b2bb8128bf182353ec5"}
Nov 02 02:11:32.618706 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.618617    5875 scope.go:122] "RemoveContainer" containerID="6488d6b44f98f66c5f2f53535711ce77f8d5e4481fabc2568b7676ead16e1365"
Nov 02 02:11:32.618957 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:11:32.618932    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6488d6b44f98f66c5f2f53535711ce77f8d5e4481fabc2568b7676ead16e1365\": container with ID starting with 6488d6b44f98f66c5f2f53535711ce77f8d5e4481fabc2568b7676ead16e1365 not found: ID does not exist" containerID="6488d6b44f98f66c5f2f53535711ce77f8d5e4481fabc2568b7676ead16e1365"
Nov 02 02:11:32.619018 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.618967    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6488d6b44f98f66c5f2f53535711ce77f8d5e4481fabc2568b7676ead16e1365"} err="failed to get container status \"6488d6b44f98f66c5f2f53535711ce77f8d5e4481fabc2568b7676ead16e1365\": rpc error: code = NotFound desc = could not find container \"6488d6b44f98f66c5f2f53535711ce77f8d5e4481fabc2568b7676ead16e1365\": container with ID starting with 6488d6b44f98f66c5f2f53535711ce77f8d5e4481fabc2568b7676ead16e1365 not found: ID does not exist"
Nov 02 02:11:32.619018 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.618990    5875 scope.go:122] "RemoveContainer" containerID="b452f792e62a5a483140cd3a1a9a34deff5e564fec59d52ad98f6175f0e9dbab"
Nov 02 02:11:32.629294 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.629274    5875 scope.go:122] "RemoveContainer" containerID="b452f792e62a5a483140cd3a1a9a34deff5e564fec59d52ad98f6175f0e9dbab"
Nov 02 02:11:32.629588 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:11:32.629571    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b452f792e62a5a483140cd3a1a9a34deff5e564fec59d52ad98f6175f0e9dbab\": container with ID starting with b452f792e62a5a483140cd3a1a9a34deff5e564fec59d52ad98f6175f0e9dbab not found: ID does not exist" containerID="b452f792e62a5a483140cd3a1a9a34deff5e564fec59d52ad98f6175f0e9dbab"
Nov 02 02:11:32.629644 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.629596    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b452f792e62a5a483140cd3a1a9a34deff5e564fec59d52ad98f6175f0e9dbab"} err="failed to get container status \"b452f792e62a5a483140cd3a1a9a34deff5e564fec59d52ad98f6175f0e9dbab\": rpc error: code = NotFound desc = could not find container \"b452f792e62a5a483140cd3a1a9a34deff5e564fec59d52ad98f6175f0e9dbab\": container with ID starting with b452f792e62a5a483140cd3a1a9a34deff5e564fec59d52ad98f6175f0e9dbab not found: ID does not exist"
Nov 02 02:11:32.629644 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.629612    5875 scope.go:122] "RemoveContainer" containerID="f19c2ef011adce2baa5a730b93da77b0ae47788c2ea6d7f0ed644a23c12de65c"
Nov 02 02:11:32.638771 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.638743    5875 scope.go:122] "RemoveContainer" containerID="f19c2ef011adce2baa5a730b93da77b0ae47788c2ea6d7f0ed644a23c12de65c"
Nov 02 02:11:32.639069 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:11:32.639047    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f19c2ef011adce2baa5a730b93da77b0ae47788c2ea6d7f0ed644a23c12de65c\": container with ID starting with f19c2ef011adce2baa5a730b93da77b0ae47788c2ea6d7f0ed644a23c12de65c not found: ID does not exist" containerID="f19c2ef011adce2baa5a730b93da77b0ae47788c2ea6d7f0ed644a23c12de65c"
Nov 02 02:11:32.639118 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.639079    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f19c2ef011adce2baa5a730b93da77b0ae47788c2ea6d7f0ed644a23c12de65c"} err="failed to get container status \"f19c2ef011adce2baa5a730b93da77b0ae47788c2ea6d7f0ed644a23c12de65c\": rpc error: code = NotFound desc = could not find container \"f19c2ef011adce2baa5a730b93da77b0ae47788c2ea6d7f0ed644a23c12de65c\": container with ID starting with f19c2ef011adce2baa5a730b93da77b0ae47788c2ea6d7f0ed644a23c12de65c not found: ID does not exist"
Nov 02 02:11:32.639118 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.639096    5875 scope.go:122] "RemoveContainer" containerID="0b62cee84c26747b0a06eaec209471c1f8029cb967b1598169949f1ef40ea83f"
Nov 02 02:11:32.648536 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.648516    5875 scope.go:122] "RemoveContainer" containerID="c06e8a7212c31d1e56abba5d488e65c1f67384f5d258f35e8f3dfa83c267712c"
Nov 02 02:11:32.650755 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.650735    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-envoy\") pod \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") "
Nov 02 02:11:32.650857 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.650789    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-podinfo\") pod \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") "
Nov 02 02:11:32.650857 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.650827    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-token\" (UniqueName: \"kubernetes.io/projected/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-token\") pod \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") "
Nov 02 02:11:32.650975 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.650860    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-data\") pod \"c764b207-bd1e-4652-b4c1-519c8f453aa2\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") "
Nov 02 02:11:32.650975 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.650913    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-token\" (UniqueName: \"kubernetes.io/projected/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-token\") pod \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") "
Nov 02 02:11:32.650975 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.650959    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istiod-ca-cert\") pod \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") "
Nov 02 02:11:32.651141 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.650987    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-workload-socket\") pod \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") "
Nov 02 02:11:32.651141 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651036    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-workload-socket\") pod \"c764b207-bd1e-4652-b4c1-519c8f453aa2\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") "
Nov 02 02:11:32.651141 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651083    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c764b207-bd1e-4652-b4c1-519c8f453aa2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c764b207-bd1e-4652-b4c1-519c8f453aa2-istiod-ca-cert\") pod \"c764b207-bd1e-4652-b4c1-519c8f453aa2\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") "
Nov 02 02:11:32.651330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651144    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istiod-ca-cert\") pod \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") "
Nov 02 02:11:32.651330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651182    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-credential-socket\") pod \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") "
Nov 02 02:11:32.651330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651199    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-data" pod "c764b207-bd1e-4652-b4c1-519c8f453aa2" (UID: "c764b207-bd1e-4652-b4c1-519c8f453aa2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.651330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651246    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-podinfo\") pod \"c764b207-bd1e-4652-b4c1-519c8f453aa2\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") "
Nov 02 02:11:32.651330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651274    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-workload-socket\") pod \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") "
Nov 02 02:11:32.651330 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651306    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-workload-certs\") pod \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") "
Nov 02 02:11:32.651650 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651345    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c764b207-bd1e-4652-b4c1-519c8f453aa2-kube-api-access-nb57p\" (UniqueName: \"kubernetes.io/projected/c764b207-bd1e-4652-b4c1-519c8f453aa2-kube-api-access-nb57p\") pod \"c764b207-bd1e-4652-b4c1-519c8f453aa2\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") "
Nov 02 02:11:32.651650 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651384    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-data\") pod \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") "
Nov 02 02:11:32.651650 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651429    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-token\" (UniqueName: \"kubernetes.io/projected/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-token\") pod \"c764b207-bd1e-4652-b4c1-519c8f453aa2\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") "
Nov 02 02:11:32.651650 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651483    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-kube-api-access-jh7qf\" (UniqueName: \"kubernetes.io/projected/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-kube-api-access-jh7qf\") pod \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") "
Nov 02 02:11:32.651650 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651542    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-envoy\") pod \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") "
Nov 02 02:11:32.651650 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651596    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-credential-socket\") pod \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") "
Nov 02 02:11:32.651650 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651633    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-kube-api-access-5j8j6\" (UniqueName: \"kubernetes.io/projected/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-kube-api-access-5j8j6\") pod \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") "
Nov 02 02:11:32.652580 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651661    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-envoy\") pod \"c764b207-bd1e-4652-b4c1-519c8f453aa2\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") "
Nov 02 02:11:32.652580 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651730    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-workload-certs\") pod \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\" (UID: \"c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611\") "
Nov 02 02:11:32.652580 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651829    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-data\") pod \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") "
Nov 02 02:11:32.652580 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651870    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-podinfo\") pod \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\" (UID: \"d03078b2-0b08-41d4-9a64-cc50f84bc2a3\") "
Nov 02 02:11:32.652580 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651899    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-credential-socket\") pod \"c764b207-bd1e-4652-b4c1-519c8f453aa2\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") "
Nov 02 02:11:32.652580 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651976    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-workload-certs\") pod \"c764b207-bd1e-4652-b4c1-519c8f453aa2\" (UID: \"c764b207-bd1e-4652-b4c1-519c8f453aa2\") "
Nov 02 02:11:32.653433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.652673    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zf26\" (UniqueName: \"kubernetes.io/projected/823b5637-9df4-4432-8c27-7de55e83c96e-kube-api-access-7zf26\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.653433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.652697    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.653433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.652710    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823b5637-9df4-4432-8c27-7de55e83c96e-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.653433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.652726    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.653433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.652740    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.653433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.652751    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.653433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.652764    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823b5637-9df4-4432-8c27-7de55e83c96e-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.653433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.652779    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823b5637-9df4-4432-8c27-7de55e83c96e-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.653433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.652795    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823b5637-9df4-4432-8c27-7de55e83c96e-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.653433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651485    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c764b207-bd1e-4652-b4c1-519c8f453aa2-istiod-ca-cert" pod "c764b207-bd1e-4652-b4c1-519c8f453aa2" (UID: "c764b207-bd1e-4652-b4c1-519c8f453aa2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:11:32.653433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651637    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-workload-socket" pod "d03078b2-0b08-41d4-9a64-cc50f84bc2a3" (UID: "d03078b2-0b08-41d4-9a64-cc50f84bc2a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.653433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651662    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-workload-socket" pod "c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611" (UID: "c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.653433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651809    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istiod-ca-cert" pod "d03078b2-0b08-41d4-9a64-cc50f84bc2a3" (UID: "d03078b2-0b08-41d4-9a64-cc50f84bc2a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:11:32.653433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.651950    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-credential-socket" pod "d03078b2-0b08-41d4-9a64-cc50f84bc2a3" (UID: "d03078b2-0b08-41d4-9a64-cc50f84bc2a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.653433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.652050    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-workload-socket" pod "c764b207-bd1e-4652-b4c1-519c8f453aa2" (UID: "c764b207-bd1e-4652-b4c1-519c8f453aa2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.653433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.652621    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-workload-certs" pod "c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611" (UID: "c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.653433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.652981    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-workload-certs" pod "c764b207-bd1e-4652-b4c1-519c8f453aa2" (UID: "c764b207-bd1e-4652-b4c1-519c8f453aa2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.653433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.653247    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-data" pod "d03078b2-0b08-41d4-9a64-cc50f84bc2a3" (UID: "d03078b2-0b08-41d4-9a64-cc50f84bc2a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.653433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.653390    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-data" pod "c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611" (UID: "c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.655150 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.653506    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-credential-socket" pod "c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611" (UID: "c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.655150 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.653566    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-workload-certs" pod "d03078b2-0b08-41d4-9a64-cc50f84bc2a3" (UID: "d03078b2-0b08-41d4-9a64-cc50f84bc2a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.655150 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.653652    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istiod-ca-cert" pod "c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611" (UID: "c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:11:32.656532 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.655137    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-credential-socket" pod "c764b207-bd1e-4652-b4c1-519c8f453aa2" (UID: "c764b207-bd1e-4652-b4c1-519c8f453aa2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.658095 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.658064    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-podinfo" pod "d03078b2-0b08-41d4-9a64-cc50f84bc2a3" (UID: "d03078b2-0b08-41d4-9a64-cc50f84bc2a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:11:32.658205 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.658095    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-envoy" pod "c764b207-bd1e-4652-b4c1-519c8f453aa2" (UID: "c764b207-bd1e-4652-b4c1-519c8f453aa2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.660162 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.660136    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c764b207-bd1e-4652-b4c1-519c8f453aa2-kube-api-access-nb57p" pod "c764b207-bd1e-4652-b4c1-519c8f453aa2" (UID: "c764b207-bd1e-4652-b4c1-519c8f453aa2"). InnerVolumeSpecName "kube-api-access-nb57p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:32.661694 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.661660    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-kube-api-access-jh7qf" pod "d03078b2-0b08-41d4-9a64-cc50f84bc2a3" (UID: "d03078b2-0b08-41d4-9a64-cc50f84bc2a3"). InnerVolumeSpecName "kube-api-access-jh7qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:32.661937 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.661795    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-kube-api-access-5j8j6" pod "c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611" (UID: "c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611"). InnerVolumeSpecName "kube-api-access-5j8j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:32.661937 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.661911    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-token" pod "c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611" (UID: "c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:32.661937 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.661918    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-envoy" pod "c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611" (UID: "c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.662199 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.661958    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-podinfo" pod "c764b207-bd1e-4652-b4c1-519c8f453aa2" (UID: "c764b207-bd1e-4652-b4c1-519c8f453aa2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:11:32.662199 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.662032    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-token" pod "d03078b2-0b08-41d4-9a64-cc50f84bc2a3" (UID: "d03078b2-0b08-41d4-9a64-cc50f84bc2a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:32.662199 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.662050    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-envoy" pod "d03078b2-0b08-41d4-9a64-cc50f84bc2a3" (UID: "d03078b2-0b08-41d4-9a64-cc50f84bc2a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:32.662199 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.662051    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-podinfo" pod "c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611" (UID: "c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:11:32.663749 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.663729    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-token" pod "c764b207-bd1e-4652-b4c1-519c8f453aa2" (UID: "c764b207-bd1e-4652-b4c1-519c8f453aa2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:32.664365 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.664340    5875 scope.go:122] "RemoveContainer" containerID="0b62cee84c26747b0a06eaec209471c1f8029cb967b1598169949f1ef40ea83f"
Nov 02 02:11:32.664756 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:11:32.664731    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b62cee84c26747b0a06eaec209471c1f8029cb967b1598169949f1ef40ea83f\": container with ID starting with 0b62cee84c26747b0a06eaec209471c1f8029cb967b1598169949f1ef40ea83f not found: ID does not exist" containerID="0b62cee84c26747b0a06eaec209471c1f8029cb967b1598169949f1ef40ea83f"
Nov 02 02:11:32.664848 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.664765    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b62cee84c26747b0a06eaec209471c1f8029cb967b1598169949f1ef40ea83f"} err="failed to get container status \"0b62cee84c26747b0a06eaec209471c1f8029cb967b1598169949f1ef40ea83f\": rpc error: code = NotFound desc = could not find container \"0b62cee84c26747b0a06eaec209471c1f8029cb967b1598169949f1ef40ea83f\": container with ID starting with 0b62cee84c26747b0a06eaec209471c1f8029cb967b1598169949f1ef40ea83f not found: ID does not exist"
Nov 02 02:11:32.664848 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.664784    5875 scope.go:122] "RemoveContainer" containerID="c06e8a7212c31d1e56abba5d488e65c1f67384f5d258f35e8f3dfa83c267712c"
Nov 02 02:11:32.665120 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:11:32.665088    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c06e8a7212c31d1e56abba5d488e65c1f67384f5d258f35e8f3dfa83c267712c\": container with ID starting with c06e8a7212c31d1e56abba5d488e65c1f67384f5d258f35e8f3dfa83c267712c not found: ID does not exist" containerID="c06e8a7212c31d1e56abba5d488e65c1f67384f5d258f35e8f3dfa83c267712c"
Nov 02 02:11:32.665185 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.665128    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c06e8a7212c31d1e56abba5d488e65c1f67384f5d258f35e8f3dfa83c267712c"} err="failed to get container status \"c06e8a7212c31d1e56abba5d488e65c1f67384f5d258f35e8f3dfa83c267712c\": rpc error: code = NotFound desc = could not find container \"c06e8a7212c31d1e56abba5d488e65c1f67384f5d258f35e8f3dfa83c267712c\": container with ID starting with c06e8a7212c31d1e56abba5d488e65c1f67384f5d258f35e8f3dfa83c267712c not found: ID does not exist"
Nov 02 02:11:32.754071 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754036    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.754071 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754073    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j8j6\" (UniqueName: \"kubernetes.io/projected/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-kube-api-access-5j8j6\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.754315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754090    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.754315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754108    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.754315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754121    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.754315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754132    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.754315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754146    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.754315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754161    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.754315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754175    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.754315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754191    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.754315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754204    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.754315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754235    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.754315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754248    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.754315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754260    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.754315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754269    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c764b207-bd1e-4652-b4c1-519c8f453aa2-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.754315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754277    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c764b207-bd1e-4652-b4c1-519c8f453aa2-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.754315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754286    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.754315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754294    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.754315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754302    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.754315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754310    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.754315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754319    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.754942 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754327    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb57p\" (UniqueName: \"kubernetes.io/projected/c764b207-bd1e-4652-b4c1-519c8f453aa2-kube-api-access-nb57p\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.754942 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754335    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.754942 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754344    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c764b207-bd1e-4652-b4c1-519c8f453aa2-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.754942 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754354    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh7qf\" (UniqueName: \"kubernetes.io/projected/d03078b2-0b08-41d4-9a64-cc50f84bc2a3-kube-api-access-jh7qf\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.754942 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.754362    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:32.771684 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.771647    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"]
Nov 02 02:11:32.775235 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:32.775201    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-wckgj"]
Nov 02 02:11:33.242363 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:33.242325    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"]
Nov 02 02:11:33.246454 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:33.246423    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-jckjs"]
Nov 02 02:11:33.355729 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:33.355693    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"]
Nov 02 02:11:33.361531 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:33.361504    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-m8ljd"]
Nov 02 02:11:33.471234 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:33.471191    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"]
Nov 02 02:11:33.476911 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:33.476883    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5700d3d1-openshift-default-98f54c69-pnr78"]
Nov 02 02:11:34.694369 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:34.694338    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="823b5637-9df4-4432-8c27-7de55e83c96e" path="/var/lib/kubelet/pods/823b5637-9df4-4432-8c27-7de55e83c96e/volumes"
Nov 02 02:11:34.694768 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:34.694754    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611" path="/var/lib/kubelet/pods/c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611/volumes"
Nov 02 02:11:34.695162 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:34.695149    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c764b207-bd1e-4652-b4c1-519c8f453aa2" path="/var/lib/kubelet/pods/c764b207-bd1e-4652-b4c1-519c8f453aa2/volumes"
Nov 02 02:11:34.695914 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:34.695898    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d03078b2-0b08-41d4-9a64-cc50f84bc2a3" path="/var/lib/kubelet/pods/d03078b2-0b08-41d4-9a64-cc50f84bc2a3/volumes"
Nov 02 02:11:35.859272 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:35.859241    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:11:35.871635 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:35.871607    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:11:36.657463 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:36.657434    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:11:38.904609 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:38.904575    5875 scope.go:122] "RemoveContainer" containerID="dd6bc2e03f40e3c3027cbe573c08b7b5134eef05070a973b40aed85d1eaab534"
Nov 02 02:11:41.319535 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:11:41.319445    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:11:56.867247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:11:56.862326    5875 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4" podUID="c30b1225-cfec-4f91-9fa0-ab84afcb0598" containerName="main" probeResult="failure" output="Get \"https://10.134.0.195:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:12:03.290379 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:03.290334    5875 patch_prober.go:29] interesting pod/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:12:03.290877 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:03.290402    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw" podUID="21514fb5-5be3-4162-bbfe-7d5a46e4a6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:12:04.881972 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:04.881942    5875 generic.go:356] "Generic (PLEG): container finished" podID="21514fb5-5be3-4162-bbfe-7d5a46e4a6f9" containerID="91ad76a6027fa6435bdfdee6f1de3b16868f36c038e5274351c150a1829f8bbc" exitCode=0
Nov 02 02:12:04.882430 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:04.882025    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw" event={"ID":"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9","Type":"ContainerDied","Data":"91ad76a6027fa6435bdfdee6f1de3b16868f36c038e5274351c150a1829f8bbc"}
Nov 02 02:12:04.882475 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:04.882466    5875 scope.go:122] "RemoveContainer" containerID="91ad76a6027fa6435bdfdee6f1de3b16868f36c038e5274351c150a1829f8bbc"
Nov 02 02:12:04.883968 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:04.883939    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:12:04.884030 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:04.884008    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:12:04.884078 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:04.884040    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:12:05.890360 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:05.890327    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw" event={"ID":"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9","Type":"ContainerStarted","Data":"5579e92da538d5d205c67730215069e4c90b2008ade6a46dc7be52f208ba4ad3"}
Nov 02 02:12:06.572732 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:06.572696    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:12:06.572921 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:06.572884    5875 patch_prober.go:29] interesting pod/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 02:12:06.572969 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:06.572926    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw" podUID="21514fb5-5be3-4162-bbfe-7d5a46e4a6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 02:12:07.572980 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:07.572946    5875 patch_prober.go:29] interesting pod/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 02:12:07.573425 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:07.573004    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw" podUID="21514fb5-5be3-4162-bbfe-7d5a46e4a6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 02:12:08.573333 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:08.573302    5875 patch_prober.go:29] interesting pod/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 02:12:08.573744 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:08.573350    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw" podUID="21514fb5-5be3-4162-bbfe-7d5a46e4a6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 02:12:09.572907 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:09.572869    5875 patch_prober.go:29] interesting pod/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 02:12:09.573107 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:09.572936    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw" podUID="21514fb5-5be3-4162-bbfe-7d5a46e4a6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 02:12:10.572627 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:10.572591    5875 patch_prober.go:29] interesting pod/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 02:12:10.573236 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:10.572651    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw" podUID="21514fb5-5be3-4162-bbfe-7d5a46e4a6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 02:12:11.319329 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:12:11.319290    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:12:11.577173 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:11.577093    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:12:11.577624 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:11.577606    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:12:11.578333 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:11.578314    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:12:14.483879 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:14.483848    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:12:21.136698 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.136666    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092079-nzrk4"]
Nov 02 02:12:21.137339 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.137323    5875 cpu_manager.go:414] "containerMap: removing container" podUID="c764b207-bd1e-4652-b4c1-519c8f453aa2" containerName="istio-proxy"
Nov 02 02:12:21.137403 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.137341    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c764b207-bd1e-4652-b4c1-519c8f453aa2" containerName="istio-proxy"
Nov 02 02:12:21.137403 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.137369    5875 cpu_manager.go:414] "containerMap: removing container" podUID="d03078b2-0b08-41d4-9a64-cc50f84bc2a3" containerName="istio-proxy"
Nov 02 02:12:21.137403 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.137375    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03078b2-0b08-41d4-9a64-cc50f84bc2a3" containerName="istio-proxy"
Nov 02 02:12:21.137403 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.137395    5875 cpu_manager.go:414] "containerMap: removing container" podUID="c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611" containerName="istio-proxy"
Nov 02 02:12:21.137403 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.137400    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611" containerName="istio-proxy"
Nov 02 02:12:21.137597 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.137414    5875 cpu_manager.go:414] "containerMap: removing container" podUID="823b5637-9df4-4432-8c27-7de55e83c96e" containerName="istio-proxy"
Nov 02 02:12:21.137597 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.137419    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823b5637-9df4-4432-8c27-7de55e83c96e" containerName="istio-proxy"
Nov 02 02:12:21.137597 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.137501    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="c764b207-bd1e-4652-b4c1-519c8f453aa2" containerName="istio-proxy"
Nov 02 02:12:21.137597 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.137513    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03078b2-0b08-41d4-9a64-cc50f84bc2a3" containerName="istio-proxy"
Nov 02 02:12:21.137597 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.137520    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="c55b5b3b-f5f4-4cbf-a7dc-38611a7bf611" containerName="istio-proxy"
Nov 02 02:12:21.137597 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.137529    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="823b5637-9df4-4432-8c27-7de55e83c96e" containerName="istio-proxy"
Nov 02 02:12:21.137597 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.137536    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="c764b207-bd1e-4652-b4c1-519c8f453aa2" containerName="istio-proxy"
Nov 02 02:12:21.141095 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.141078    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092079-nzrk4"
Nov 02 02:12:21.192818 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.192793    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:12:21.192979 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.192832    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:12:21.206548 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.206515    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqb5z\" (UniqueName: \"kubernetes.io/projected/93c193d8-2d88-4ee4-8682-c654362e27e1-kube-api-access-hqb5z\") pod \"maas-api-revoke-keys-e2e-api-a-092079-nzrk4\" (UID: \"93c193d8-2d88-4ee4-8682-c654362e27e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092079-nzrk4"
Nov 02 02:12:21.206810 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.206786    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c193d8-2d88-4ee4-8682-c654362e27e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-092079-nzrk4\" (UID: \"93c193d8-2d88-4ee4-8682-c654362e27e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092079-nzrk4"
Nov 02 02:12:21.206871 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.206840    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c193d8-2d88-4ee4-8682-c654362e27e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-092079-nzrk4\" (UID: \"93c193d8-2d88-4ee4-8682-c654362e27e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092079-nzrk4"
Nov 02 02:12:21.260941 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.260868    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092079-nzrk4"]
Nov 02 02:12:21.308132 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.308102    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqb5z\" (UniqueName: \"kubernetes.io/projected/93c193d8-2d88-4ee4-8682-c654362e27e1-kube-api-access-hqb5z\") pod \"maas-api-revoke-keys-e2e-api-a-092079-nzrk4\" (UID: \"93c193d8-2d88-4ee4-8682-c654362e27e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092079-nzrk4"
Nov 02 02:12:21.308343 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.308213    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c193d8-2d88-4ee4-8682-c654362e27e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-092079-nzrk4\" (UID: \"93c193d8-2d88-4ee4-8682-c654362e27e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092079-nzrk4"
Nov 02 02:12:21.308343 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.308291    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c193d8-2d88-4ee4-8682-c654362e27e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-092079-nzrk4\" (UID: \"93c193d8-2d88-4ee4-8682-c654362e27e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092079-nzrk4"
Nov 02 02:12:21.308802 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.308779    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c193d8-2d88-4ee4-8682-c654362e27e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-092079-nzrk4\" (UID: \"93c193d8-2d88-4ee4-8682-c654362e27e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092079-nzrk4"
Nov 02 02:12:21.310681 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.310662    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c193d8-2d88-4ee4-8682-c654362e27e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-092079-nzrk4\" (UID: \"93c193d8-2d88-4ee4-8682-c654362e27e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092079-nzrk4"
Nov 02 02:12:21.368823 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.368781    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqb5z\" (UniqueName: \"kubernetes.io/projected/93c193d8-2d88-4ee4-8682-c654362e27e1-kube-api-access-hqb5z\") pod \"maas-api-revoke-keys-e2e-api-a-092079-nzrk4\" (UID: \"93c193d8-2d88-4ee4-8682-c654362e27e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092079-nzrk4"
Nov 02 02:12:21.454452 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.454380    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092079-nzrk4"
Nov 02 02:12:21.828250 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:21.828201    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092079-nzrk4"]
Nov 02 02:12:21.829909 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:12:21.829883    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c193d8_2d88_4ee4_8682_c654362e27e1.slice/crio-2a6a01b16b77dd86203dd393fe2f9dc7295bdf7cf205ef97cfba5c35ed048107 WatchSource:0}: Error finding container 2a6a01b16b77dd86203dd393fe2f9dc7295bdf7cf205ef97cfba5c35ed048107: Status 404 returned error can't find the container with id 2a6a01b16b77dd86203dd393fe2f9dc7295bdf7cf205ef97cfba5c35ed048107
Nov 02 02:12:22.017694 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:22.017663    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092079-nzrk4" event={"ID":"93c193d8-2d88-4ee4-8682-c654362e27e1","Type":"ContainerStarted","Data":"2a6a01b16b77dd86203dd393fe2f9dc7295bdf7cf205ef97cfba5c35ed048107"}
Nov 02 02:12:23.025482 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:23.025446    5875 generic.go:356] "Generic (PLEG): container finished" podID="93c193d8-2d88-4ee4-8682-c654362e27e1" containerID="1d4ccf17b1f934bdd4358ee3c2201ace5745857905b2279c3fc7b0b13f5305a5" exitCode=0
Nov 02 02:12:23.025914 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:23.025539    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092079-nzrk4" event={"ID":"93c193d8-2d88-4ee4-8682-c654362e27e1","Type":"ContainerDied","Data":"1d4ccf17b1f934bdd4358ee3c2201ace5745857905b2279c3fc7b0b13f5305a5"}
Nov 02 02:12:24.171825 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:24.171798    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092079-nzrk4"
Nov 02 02:12:24.244957 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:24.244929    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c193d8-2d88-4ee4-8682-c654362e27e1-kube-api-access-hqb5z\" (UniqueName: \"kubernetes.io/projected/93c193d8-2d88-4ee4-8682-c654362e27e1-kube-api-access-hqb5z\") pod \"93c193d8-2d88-4ee4-8682-c654362e27e1\" (UID: \"93c193d8-2d88-4ee4-8682-c654362e27e1\") "
Nov 02 02:12:24.245145 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:24.244978    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c193d8-2d88-4ee4-8682-c654362e27e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c193d8-2d88-4ee4-8682-c654362e27e1-maas-api-openshift-ca\") pod \"93c193d8-2d88-4ee4-8682-c654362e27e1\" (UID: \"93c193d8-2d88-4ee4-8682-c654362e27e1\") "
Nov 02 02:12:24.245145 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:24.245117    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c193d8-2d88-4ee4-8682-c654362e27e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c193d8-2d88-4ee4-8682-c654362e27e1-maas-api-xks-ca\") pod \"93c193d8-2d88-4ee4-8682-c654362e27e1\" (UID: \"93c193d8-2d88-4ee4-8682-c654362e27e1\") "
Nov 02 02:12:24.245416 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:24.245383    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c193d8-2d88-4ee4-8682-c654362e27e1-maas-api-openshift-ca" pod "93c193d8-2d88-4ee4-8682-c654362e27e1" (UID: "93c193d8-2d88-4ee4-8682-c654362e27e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:12:24.245627 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:24.245606    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c193d8-2d88-4ee4-8682-c654362e27e1-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:24.247284 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:24.247252    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c193d8-2d88-4ee4-8682-c654362e27e1-maas-api-xks-ca" pod "93c193d8-2d88-4ee4-8682-c654362e27e1" (UID: "93c193d8-2d88-4ee4-8682-c654362e27e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:12:24.247390 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:24.247334    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c193d8-2d88-4ee4-8682-c654362e27e1-kube-api-access-hqb5z" pod "93c193d8-2d88-4ee4-8682-c654362e27e1" (UID: "93c193d8-2d88-4ee4-8682-c654362e27e1"). InnerVolumeSpecName "kube-api-access-hqb5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:12:24.347105 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:24.347035    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c193d8-2d88-4ee4-8682-c654362e27e1-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:24.347105 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:24.347064    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqb5z\" (UniqueName: \"kubernetes.io/projected/93c193d8-2d88-4ee4-8682-c654362e27e1-kube-api-access-hqb5z\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:25.038751 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.038716    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092079-nzrk4"
Nov 02 02:12:25.038968 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.038747    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092079-nzrk4" event={"ID":"93c193d8-2d88-4ee4-8682-c654362e27e1","Type":"ContainerDied","Data":"2a6a01b16b77dd86203dd393fe2f9dc7295bdf7cf205ef97cfba5c35ed048107"}
Nov 02 02:12:25.038968 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.038795    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a6a01b16b77dd86203dd393fe2f9dc7295bdf7cf205ef97cfba5c35ed048107"
Nov 02 02:12:25.166312 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.166276    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5"]
Nov 02 02:12:25.166960 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.166940    5875 cpu_manager.go:414] "containerMap: removing container" podUID="93c193d8-2d88-4ee4-8682-c654362e27e1" containerName="revoke-keys"
Nov 02 02:12:25.166960 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.166959    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c193d8-2d88-4ee4-8682-c654362e27e1" containerName="revoke-keys"
Nov 02 02:12:25.167067 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.166999    5875 cpu_manager.go:414] "containerMap: removing container" podUID="c764b207-bd1e-4652-b4c1-519c8f453aa2" containerName="istio-proxy"
Nov 02 02:12:25.167067 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.167004    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c764b207-bd1e-4652-b4c1-519c8f453aa2" containerName="istio-proxy"
Nov 02 02:12:25.167147 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.167092    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c193d8-2d88-4ee4-8682-c654362e27e1" containerName="revoke-keys"
Nov 02 02:12:25.170427 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.170406    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5"
Nov 02 02:12:25.225396 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.225347    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:12:25.225869 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.225419    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:12:25.228125 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.228100    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5"]
Nov 02 02:12:25.258208 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.258167    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cs5t\" (UniqueName: \"kubernetes.io/projected/5641707e-23f7-4caa-b9cd-485037333d2c-kube-api-access-7cs5t\") pod \"maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5\" (UID: \"5641707e-23f7-4caa-b9cd-485037333d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5"
Nov 02 02:12:25.258365 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.258299    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5641707e-23f7-4caa-b9cd-485037333d2c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5\" (UID: \"5641707e-23f7-4caa-b9cd-485037333d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5"
Nov 02 02:12:25.258410 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.258378    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5641707e-23f7-4caa-b9cd-485037333d2c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5\" (UID: \"5641707e-23f7-4caa-b9cd-485037333d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5"
Nov 02 02:12:25.360088 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.360005    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cs5t\" (UniqueName: \"kubernetes.io/projected/5641707e-23f7-4caa-b9cd-485037333d2c-kube-api-access-7cs5t\") pod \"maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5\" (UID: \"5641707e-23f7-4caa-b9cd-485037333d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5"
Nov 02 02:12:25.360088 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.360057    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5641707e-23f7-4caa-b9cd-485037333d2c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5\" (UID: \"5641707e-23f7-4caa-b9cd-485037333d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5"
Nov 02 02:12:25.360319 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.360093    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5641707e-23f7-4caa-b9cd-485037333d2c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5\" (UID: \"5641707e-23f7-4caa-b9cd-485037333d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5"
Nov 02 02:12:25.360755 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.360712    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5641707e-23f7-4caa-b9cd-485037333d2c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5\" (UID: \"5641707e-23f7-4caa-b9cd-485037333d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5"
Nov 02 02:12:25.362594 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.362572    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5641707e-23f7-4caa-b9cd-485037333d2c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5\" (UID: \"5641707e-23f7-4caa-b9cd-485037333d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5"
Nov 02 02:12:25.423315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.423288    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cs5t\" (UniqueName: \"kubernetes.io/projected/5641707e-23f7-4caa-b9cd-485037333d2c-kube-api-access-7cs5t\") pod \"maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5\" (UID: \"5641707e-23f7-4caa-b9cd-485037333d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5"
Nov 02 02:12:25.484608 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.484575    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5"
Nov 02 02:12:25.533581 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.533551    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aab23e3f-lgsvs"]
Nov 02 02:12:25.537839 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.537818    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab23e3f-lgsvs"
Nov 02 02:12:25.594637 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.594581    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aab23e3f-lgsvs"]
Nov 02 02:12:25.663750 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.663673    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34647927-454c-4bb5-8c38-70c2502839e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aab23e3f-lgsvs\" (UID: \"34647927-454c-4bb5-8c38-70c2502839e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab23e3f-lgsvs"
Nov 02 02:12:25.663920 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.663827    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34647927-454c-4bb5-8c38-70c2502839e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aab23e3f-lgsvs\" (UID: \"34647927-454c-4bb5-8c38-70c2502839e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab23e3f-lgsvs"
Nov 02 02:12:25.663988 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.663969    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csjrl\" (UniqueName: \"kubernetes.io/projected/34647927-454c-4bb5-8c38-70c2502839e7-kube-api-access-csjrl\") pod \"maas-api-revoke-sub-aab23e3f-lgsvs\" (UID: \"34647927-454c-4bb5-8c38-70c2502839e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab23e3f-lgsvs"
Nov 02 02:12:25.764898 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.764871    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csjrl\" (UniqueName: \"kubernetes.io/projected/34647927-454c-4bb5-8c38-70c2502839e7-kube-api-access-csjrl\") pod \"maas-api-revoke-sub-aab23e3f-lgsvs\" (UID: \"34647927-454c-4bb5-8c38-70c2502839e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab23e3f-lgsvs"
Nov 02 02:12:25.765061 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.764966    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34647927-454c-4bb5-8c38-70c2502839e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aab23e3f-lgsvs\" (UID: \"34647927-454c-4bb5-8c38-70c2502839e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab23e3f-lgsvs"
Nov 02 02:12:25.765117 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.765060    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34647927-454c-4bb5-8c38-70c2502839e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aab23e3f-lgsvs\" (UID: \"34647927-454c-4bb5-8c38-70c2502839e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab23e3f-lgsvs"
Nov 02 02:12:25.765691 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.765665    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34647927-454c-4bb5-8c38-70c2502839e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aab23e3f-lgsvs\" (UID: \"34647927-454c-4bb5-8c38-70c2502839e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab23e3f-lgsvs"
Nov 02 02:12:25.767446 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.767425    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34647927-454c-4bb5-8c38-70c2502839e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aab23e3f-lgsvs\" (UID: \"34647927-454c-4bb5-8c38-70c2502839e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab23e3f-lgsvs"
Nov 02 02:12:25.825022 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.824994    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csjrl\" (UniqueName: \"kubernetes.io/projected/34647927-454c-4bb5-8c38-70c2502839e7-kube-api-access-csjrl\") pod \"maas-api-revoke-sub-aab23e3f-lgsvs\" (UID: \"34647927-454c-4bb5-8c38-70c2502839e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab23e3f-lgsvs"
Nov 02 02:12:25.858306 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.858284    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab23e3f-lgsvs"
Nov 02 02:12:25.863504 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:25.863479    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5"]
Nov 02 02:12:25.869597 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:12:25.869574    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5641707e_23f7_4caa_b9cd_485037333d2c.slice/crio-7d89cf270d654ef0fca4573fc131c53e0d545118f850f88e6381d0dee3718e3f WatchSource:0}: Error finding container 7d89cf270d654ef0fca4573fc131c53e0d545118f850f88e6381d0dee3718e3f: Status 404 returned error can't find the container with id 7d89cf270d654ef0fca4573fc131c53e0d545118f850f88e6381d0dee3718e3f
Nov 02 02:12:26.046277 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:26.046245    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5" event={"ID":"5641707e-23f7-4caa-b9cd-485037333d2c","Type":"ContainerStarted","Data":"7d89cf270d654ef0fca4573fc131c53e0d545118f850f88e6381d0dee3718e3f"}
Nov 02 02:12:26.240236 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:26.240193    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aab23e3f-lgsvs"]
Nov 02 02:12:26.241277 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:12:26.241254    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34647927_454c_4bb5_8c38_70c2502839e7.slice/crio-2610332ff889b7b3dfbacd99fce8a6c12dab1236347c8e40bca05d5c2c9b3abb WatchSource:0}: Error finding container 2610332ff889b7b3dfbacd99fce8a6c12dab1236347c8e40bca05d5c2c9b3abb: Status 404 returned error can't find the container with id 2610332ff889b7b3dfbacd99fce8a6c12dab1236347c8e40bca05d5c2c9b3abb
Nov 02 02:12:27.054731 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:27.054642    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab23e3f-lgsvs" event={"ID":"34647927-454c-4bb5-8c38-70c2502839e7","Type":"ContainerStarted","Data":"6546f60fc70f6cd9004944ec1db41f61fe71aeaf667f46a0dd184c0b38fd7540"}
Nov 02 02:12:27.054731 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:27.054707    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab23e3f-lgsvs" event={"ID":"34647927-454c-4bb5-8c38-70c2502839e7","Type":"ContainerStarted","Data":"2610332ff889b7b3dfbacd99fce8a6c12dab1236347c8e40bca05d5c2c9b3abb"}
Nov 02 02:12:27.056313 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:27.056283    5875 generic.go:356] "Generic (PLEG): container finished" podID="5641707e-23f7-4caa-b9cd-485037333d2c" containerID="1943be5bbb0f2e77c63fb9dadd986b2623d9a8a96366f148b074ee010a1943ec" exitCode=0
Nov 02 02:12:27.056466 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:27.056329    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5" event={"ID":"5641707e-23f7-4caa-b9cd-485037333d2c","Type":"ContainerDied","Data":"1943be5bbb0f2e77c63fb9dadd986b2623d9a8a96366f148b074ee010a1943ec"}
Nov 02 02:12:27.170242 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:27.170170    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab23e3f-lgsvs" podStartSLOduration=2.170157047 podStartE2EDuration="2.170157047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:12:27.169308846 +0000 UTC m=+2394.610345344" watchObservedRunningTime="2026-11-02 02:12:27.170157047 +0000 UTC m=+2394.611193543"
Nov 02 02:12:28.065128 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:28.065094    5875 generic.go:356] "Generic (PLEG): container finished" podID="34647927-454c-4bb5-8c38-70c2502839e7" containerID="6546f60fc70f6cd9004944ec1db41f61fe71aeaf667f46a0dd184c0b38fd7540" exitCode=0
Nov 02 02:12:28.065590 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:28.065168    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab23e3f-lgsvs" event={"ID":"34647927-454c-4bb5-8c38-70c2502839e7","Type":"ContainerDied","Data":"6546f60fc70f6cd9004944ec1db41f61fe71aeaf667f46a0dd184c0b38fd7540"}
Nov 02 02:12:28.211375 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:28.211346    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5"
Nov 02 02:12:28.298336 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:28.298307    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5641707e-23f7-4caa-b9cd-485037333d2c-kube-api-access-7cs5t\" (UniqueName: \"kubernetes.io/projected/5641707e-23f7-4caa-b9cd-485037333d2c-kube-api-access-7cs5t\") pod \"5641707e-23f7-4caa-b9cd-485037333d2c\" (UID: \"5641707e-23f7-4caa-b9cd-485037333d2c\") "
Nov 02 02:12:28.298505 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:28.298384    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5641707e-23f7-4caa-b9cd-485037333d2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5641707e-23f7-4caa-b9cd-485037333d2c-maas-api-openshift-ca\") pod \"5641707e-23f7-4caa-b9cd-485037333d2c\" (UID: \"5641707e-23f7-4caa-b9cd-485037333d2c\") "
Nov 02 02:12:28.298505 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:28.298476    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5641707e-23f7-4caa-b9cd-485037333d2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5641707e-23f7-4caa-b9cd-485037333d2c-maas-api-xks-ca\") pod \"5641707e-23f7-4caa-b9cd-485037333d2c\" (UID: \"5641707e-23f7-4caa-b9cd-485037333d2c\") "
Nov 02 02:12:28.298777 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:28.298751    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5641707e-23f7-4caa-b9cd-485037333d2c-maas-api-openshift-ca" pod "5641707e-23f7-4caa-b9cd-485037333d2c" (UID: "5641707e-23f7-4caa-b9cd-485037333d2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:12:28.300507 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:28.300483    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5641707e-23f7-4caa-b9cd-485037333d2c-kube-api-access-7cs5t" pod "5641707e-23f7-4caa-b9cd-485037333d2c" (UID: "5641707e-23f7-4caa-b9cd-485037333d2c"). InnerVolumeSpecName "kube-api-access-7cs5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:12:28.300621 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:28.300576    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5641707e-23f7-4caa-b9cd-485037333d2c-maas-api-xks-ca" pod "5641707e-23f7-4caa-b9cd-485037333d2c" (UID: "5641707e-23f7-4caa-b9cd-485037333d2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:12:28.399962 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:28.399884    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cs5t\" (UniqueName: \"kubernetes.io/projected/5641707e-23f7-4caa-b9cd-485037333d2c-kube-api-access-7cs5t\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:28.399962 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:28.399912    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5641707e-23f7-4caa-b9cd-485037333d2c-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:28.399962 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:28.399923    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5641707e-23f7-4caa-b9cd-485037333d2c-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:29.072995 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:29.072964    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5"
Nov 02 02:12:29.073581 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:29.073012    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5" event={"ID":"5641707e-23f7-4caa-b9cd-485037333d2c","Type":"ContainerDied","Data":"7d89cf270d654ef0fca4573fc131c53e0d545118f850f88e6381d0dee3718e3f"}
Nov 02 02:12:29.073581 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:29.073072    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d89cf270d654ef0fca4573fc131c53e0d545118f850f88e6381d0dee3718e3f"
Nov 02 02:12:29.212079 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:29.212057    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab23e3f-lgsvs"
Nov 02 02:12:29.309908 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:29.309875    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34647927-454c-4bb5-8c38-70c2502839e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34647927-454c-4bb5-8c38-70c2502839e7-maas-api-xks-ca\") pod \"34647927-454c-4bb5-8c38-70c2502839e7\" (UID: \"34647927-454c-4bb5-8c38-70c2502839e7\") "
Nov 02 02:12:29.310051 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:29.309966    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34647927-454c-4bb5-8c38-70c2502839e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34647927-454c-4bb5-8c38-70c2502839e7-maas-api-openshift-ca\") pod \"34647927-454c-4bb5-8c38-70c2502839e7\" (UID: \"34647927-454c-4bb5-8c38-70c2502839e7\") "
Nov 02 02:12:29.310089 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:29.310055    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34647927-454c-4bb5-8c38-70c2502839e7-kube-api-access-csjrl\" (UniqueName: \"kubernetes.io/projected/34647927-454c-4bb5-8c38-70c2502839e7-kube-api-access-csjrl\") pod \"34647927-454c-4bb5-8c38-70c2502839e7\" (UID: \"34647927-454c-4bb5-8c38-70c2502839e7\") "
Nov 02 02:12:29.310333 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:29.310308    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34647927-454c-4bb5-8c38-70c2502839e7-maas-api-openshift-ca" pod "34647927-454c-4bb5-8c38-70c2502839e7" (UID: "34647927-454c-4bb5-8c38-70c2502839e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:12:29.310554 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:29.310533    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34647927-454c-4bb5-8c38-70c2502839e7-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:29.312122 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:29.312103    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34647927-454c-4bb5-8c38-70c2502839e7-maas-api-xks-ca" pod "34647927-454c-4bb5-8c38-70c2502839e7" (UID: "34647927-454c-4bb5-8c38-70c2502839e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:12:29.312249 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:29.312215    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34647927-454c-4bb5-8c38-70c2502839e7-kube-api-access-csjrl" pod "34647927-454c-4bb5-8c38-70c2502839e7" (UID: "34647927-454c-4bb5-8c38-70c2502839e7"). InnerVolumeSpecName "kube-api-access-csjrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:12:29.411800 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:29.411726    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csjrl\" (UniqueName: \"kubernetes.io/projected/34647927-454c-4bb5-8c38-70c2502839e7-kube-api-access-csjrl\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:29.411800 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:29.411755    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34647927-454c-4bb5-8c38-70c2502839e7-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:29.652096 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:29.652063    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"]
Nov 02 02:12:30.081722 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:30.081693    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab23e3f-lgsvs"
Nov 02 02:12:30.081722 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:30.081713    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aab23e3f-lgsvs" event={"ID":"34647927-454c-4bb5-8c38-70c2502839e7","Type":"ContainerDied","Data":"2610332ff889b7b3dfbacd99fce8a6c12dab1236347c8e40bca05d5c2c9b3abb"}
Nov 02 02:12:30.082165 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:30.081776    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2610332ff889b7b3dfbacd99fce8a6c12dab1236347c8e40bca05d5c2c9b3abb"
Nov 02 02:12:32.604621 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:32.604588    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-092079-9c6765d77-fxqv2"]
Nov 02 02:12:32.709282 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:32.709249    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s"]
Nov 02 02:12:36.694171 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:36.694136    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d"]
Nov 02 02:12:36.694753 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:36.694738    5875 cpu_manager.go:414] "containerMap: removing container" podUID="34647927-454c-4bb5-8c38-70c2502839e7" containerName="revoke-keys"
Nov 02 02:12:36.694827 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:36.694755    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34647927-454c-4bb5-8c38-70c2502839e7" containerName="revoke-keys"
Nov 02 02:12:36.694827 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:36.694769    5875 cpu_manager.go:414] "containerMap: removing container" podUID="5641707e-23f7-4caa-b9cd-485037333d2c" containerName="revoke-keys"
Nov 02 02:12:36.694827 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:36.694774    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5641707e-23f7-4caa-b9cd-485037333d2c" containerName="revoke-keys"
Nov 02 02:12:36.694948 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:36.694864    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="34647927-454c-4bb5-8c38-70c2502839e7" containerName="revoke-keys"
Nov 02 02:12:36.694948 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:36.694874    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="5641707e-23f7-4caa-b9cd-485037333d2c" containerName="revoke-keys"
Nov 02 02:12:36.698536 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:36.698519    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d"
Nov 02 02:12:36.704883 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:36.704855    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2a2a075-44e5-47a4-bcbf-f04d16246a35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d\" (UID: \"b2a2a075-44e5-47a4-bcbf-f04d16246a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d"
Nov 02 02:12:36.704993 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:36.704900    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2a2a075-44e5-47a4-bcbf-f04d16246a35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d\" (UID: \"b2a2a075-44e5-47a4-bcbf-f04d16246a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d"
Nov 02 02:12:36.705114 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:36.705095    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9mzt\" (UniqueName: \"kubernetes.io/projected/b2a2a075-44e5-47a4-bcbf-f04d16246a35-kube-api-access-p9mzt\") pod \"maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d\" (UID: \"b2a2a075-44e5-47a4-bcbf-f04d16246a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d"
Nov 02 02:12:36.749507 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:36.749480    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d"]
Nov 02 02:12:36.750409 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:36.750359    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:12:36.795818 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:36.795789    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:12:36.806532 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:36.806506    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2a2a075-44e5-47a4-bcbf-f04d16246a35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d\" (UID: \"b2a2a075-44e5-47a4-bcbf-f04d16246a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d"
Nov 02 02:12:36.806637 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:36.806553    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2a2a075-44e5-47a4-bcbf-f04d16246a35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d\" (UID: \"b2a2a075-44e5-47a4-bcbf-f04d16246a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d"
Nov 02 02:12:36.806702 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:36.806691    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9mzt\" (UniqueName: \"kubernetes.io/projected/b2a2a075-44e5-47a4-bcbf-f04d16246a35-kube-api-access-p9mzt\") pod \"maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d\" (UID: \"b2a2a075-44e5-47a4-bcbf-f04d16246a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d"
Nov 02 02:12:36.807323 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:36.807303    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2a2a075-44e5-47a4-bcbf-f04d16246a35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d\" (UID: \"b2a2a075-44e5-47a4-bcbf-f04d16246a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d"
Nov 02 02:12:36.809062 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:36.809038    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2a2a075-44e5-47a4-bcbf-f04d16246a35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d\" (UID: \"b2a2a075-44e5-47a4-bcbf-f04d16246a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d"
Nov 02 02:12:36.867026 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:36.867002    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9mzt\" (UniqueName: \"kubernetes.io/projected/b2a2a075-44e5-47a4-bcbf-f04d16246a35-kube-api-access-p9mzt\") pod \"maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d\" (UID: \"b2a2a075-44e5-47a4-bcbf-f04d16246a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d"
Nov 02 02:12:37.012152 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:37.012125    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d"
Nov 02 02:12:37.388974 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:37.388946    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d"]
Nov 02 02:12:37.390535 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:12:37.390503    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a2a075_44e5_47a4_bcbf_f04d16246a35.slice/crio-5968c2cde8871fd83d8f9595d2c1fb158ed542b3e7ed8702dd7028d323c2c4ec WatchSource:0}: Error finding container 5968c2cde8871fd83d8f9595d2c1fb158ed542b3e7ed8702dd7028d323c2c4ec: Status 404 returned error can't find the container with id 5968c2cde8871fd83d8f9595d2c1fb158ed542b3e7ed8702dd7028d323c2c4ec
Nov 02 02:12:37.605533 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:37.605493    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-092079-9c6765d77-fxqv2" podUID="c5ae28f8-e723-4bb3-b368-6fb9881d24bf" containerName="payload-processing" containerID="cri-o://ef9a96fe1b24c4129f691c59adf53af988aea38b4ff3e03ee51d39c951b5c131" gracePeriod=25
Nov 02 02:12:37.710012 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:37.709976    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s" podUID="051907ab-4e37-4298-9291-0df5fa65d735" containerName="payload-pre-processing" containerID="cri-o://2b3cde8cadc4e23ff6dc01b40b8bb2bbe4bacf8fee07c566561afd211d1d3097" gracePeriod=25
Nov 02 02:12:37.840441 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:37.840413    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4"]
Nov 02 02:12:37.866405 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:37.866378    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-092079-9c6765d77-fxqv2"
Nov 02 02:12:37.920162 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:37.920129    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5ae28f8-e723-4bb3-b368-6fb9881d24bf-kube-api-access-bfx9l\" (UniqueName: \"kubernetes.io/projected/c5ae28f8-e723-4bb3-b368-6fb9881d24bf-kube-api-access-bfx9l\") pod \"c5ae28f8-e723-4bb3-b368-6fb9881d24bf\" (UID: \"c5ae28f8-e723-4bb3-b368-6fb9881d24bf\") "
Nov 02 02:12:37.920385 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:37.920259    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5ae28f8-e723-4bb3-b368-6fb9881d24bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ae28f8-e723-4bb3-b368-6fb9881d24bf-plugins-config-volume\") pod \"c5ae28f8-e723-4bb3-b368-6fb9881d24bf\" (UID: \"c5ae28f8-e723-4bb3-b368-6fb9881d24bf\") "
Nov 02 02:12:37.920698 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:37.920673    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5ae28f8-e723-4bb3-b368-6fb9881d24bf-plugins-config-volume" pod "c5ae28f8-e723-4bb3-b368-6fb9881d24bf" (UID: "c5ae28f8-e723-4bb3-b368-6fb9881d24bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:12:37.920828 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:37.920811    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5ae28f8-e723-4bb3-b368-6fb9881d24bf-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:37.925724 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:37.925678    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ae28f8-e723-4bb3-b368-6fb9881d24bf-kube-api-access-bfx9l" pod "c5ae28f8-e723-4bb3-b368-6fb9881d24bf" (UID: "c5ae28f8-e723-4bb3-b368-6fb9881d24bf"). InnerVolumeSpecName "kube-api-access-bfx9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:12:37.990958 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:37.990930    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s"
Nov 02 02:12:38.022106 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.022080    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/051907ab-4e37-4298-9291-0df5fa65d735-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/051907ab-4e37-4298-9291-0df5fa65d735-plugins-config-volume\") pod \"051907ab-4e37-4298-9291-0df5fa65d735\" (UID: \"051907ab-4e37-4298-9291-0df5fa65d735\") "
Nov 02 02:12:38.022295 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.022113    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/051907ab-4e37-4298-9291-0df5fa65d735-kube-api-access-g6pvv\" (UniqueName: \"kubernetes.io/projected/051907ab-4e37-4298-9291-0df5fa65d735-kube-api-access-g6pvv\") pod \"051907ab-4e37-4298-9291-0df5fa65d735\" (UID: \"051907ab-4e37-4298-9291-0df5fa65d735\") "
Nov 02 02:12:38.022489 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.022475    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfx9l\" (UniqueName: \"kubernetes.io/projected/c5ae28f8-e723-4bb3-b368-6fb9881d24bf-kube-api-access-bfx9l\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:38.022545 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.022522    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051907ab-4e37-4298-9291-0df5fa65d735-plugins-config-volume" pod "051907ab-4e37-4298-9291-0df5fa65d735" (UID: "051907ab-4e37-4298-9291-0df5fa65d735"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:12:38.024393 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.024374    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051907ab-4e37-4298-9291-0df5fa65d735-kube-api-access-g6pvv" pod "051907ab-4e37-4298-9291-0df5fa65d735" (UID: "051907ab-4e37-4298-9291-0df5fa65d735"). InnerVolumeSpecName "kube-api-access-g6pvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:12:38.123386 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.123357    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/051907ab-4e37-4298-9291-0df5fa65d735-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:38.123386 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.123383    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6pvv\" (UniqueName: \"kubernetes.io/projected/051907ab-4e37-4298-9291-0df5fa65d735-kube-api-access-g6pvv\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:38.133833 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.133796    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d" event={"ID":"b2a2a075-44e5-47a4-bcbf-f04d16246a35","Type":"ContainerStarted","Data":"29c493e5452b5dffc7b7bf20fe44fee30d3e8550582be8a75737f874b4e78eb6"}
Nov 02 02:12:38.133993 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.133864    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d" event={"ID":"b2a2a075-44e5-47a4-bcbf-f04d16246a35","Type":"ContainerStarted","Data":"5968c2cde8871fd83d8f9595d2c1fb158ed542b3e7ed8702dd7028d323c2c4ec"}
Nov 02 02:12:38.135112 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.135088    5875 generic.go:356] "Generic (PLEG): container finished" podID="c5ae28f8-e723-4bb3-b368-6fb9881d24bf" containerID="ef9a96fe1b24c4129f691c59adf53af988aea38b4ff3e03ee51d39c951b5c131" exitCode=0
Nov 02 02:12:38.135275 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.135129    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-092079-9c6765d77-fxqv2"
Nov 02 02:12:38.135275 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.135170    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-092079-9c6765d77-fxqv2" event={"ID":"c5ae28f8-e723-4bb3-b368-6fb9881d24bf","Type":"ContainerDied","Data":"ef9a96fe1b24c4129f691c59adf53af988aea38b4ff3e03ee51d39c951b5c131"}
Nov 02 02:12:38.135275 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.135211    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-092079-9c6765d77-fxqv2" event={"ID":"c5ae28f8-e723-4bb3-b368-6fb9881d24bf","Type":"ContainerDied","Data":"ed839cd145e72fc48f59945d576670745b633715b4c3c84953e2a210886312b3"}
Nov 02 02:12:38.135275 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.135266    5875 scope.go:122] "RemoveContainer" containerID="ef9a96fe1b24c4129f691c59adf53af988aea38b4ff3e03ee51d39c951b5c131"
Nov 02 02:12:38.137085 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.137058    5875 generic.go:356] "Generic (PLEG): container finished" podID="051907ab-4e37-4298-9291-0df5fa65d735" containerID="2b3cde8cadc4e23ff6dc01b40b8bb2bbe4bacf8fee07c566561afd211d1d3097" exitCode=0
Nov 02 02:12:38.137183 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.137108    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s" event={"ID":"051907ab-4e37-4298-9291-0df5fa65d735","Type":"ContainerDied","Data":"2b3cde8cadc4e23ff6dc01b40b8bb2bbe4bacf8fee07c566561afd211d1d3097"}
Nov 02 02:12:38.137183 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.137124    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s"
Nov 02 02:12:38.137513 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.137143    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s" event={"ID":"051907ab-4e37-4298-9291-0df5fa65d735","Type":"ContainerDied","Data":"7e4a69901c0ca39b1956b09c058511b041e605d30b8d1f4cfa3f8b9c4be67d03"}
Nov 02 02:12:38.150200 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.150115    5875 scope.go:122] "RemoveContainer" containerID="ef9a96fe1b24c4129f691c59adf53af988aea38b4ff3e03ee51d39c951b5c131"
Nov 02 02:12:38.150724 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:12:38.150694    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef9a96fe1b24c4129f691c59adf53af988aea38b4ff3e03ee51d39c951b5c131\": container with ID starting with ef9a96fe1b24c4129f691c59adf53af988aea38b4ff3e03ee51d39c951b5c131 not found: ID does not exist" containerID="ef9a96fe1b24c4129f691c59adf53af988aea38b4ff3e03ee51d39c951b5c131"
Nov 02 02:12:38.151051 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.150961    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef9a96fe1b24c4129f691c59adf53af988aea38b4ff3e03ee51d39c951b5c131"} err="failed to get container status \"ef9a96fe1b24c4129f691c59adf53af988aea38b4ff3e03ee51d39c951b5c131\": rpc error: code = NotFound desc = could not find container \"ef9a96fe1b24c4129f691c59adf53af988aea38b4ff3e03ee51d39c951b5c131\": container with ID starting with ef9a96fe1b24c4129f691c59adf53af988aea38b4ff3e03ee51d39c951b5c131 not found: ID does not exist"
Nov 02 02:12:38.151051 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.150998    5875 scope.go:122] "RemoveContainer" containerID="2b3cde8cadc4e23ff6dc01b40b8bb2bbe4bacf8fee07c566561afd211d1d3097"
Nov 02 02:12:38.165036 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.165010    5875 scope.go:122] "RemoveContainer" containerID="2b3cde8cadc4e23ff6dc01b40b8bb2bbe4bacf8fee07c566561afd211d1d3097"
Nov 02 02:12:38.165532 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:12:38.165493    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b3cde8cadc4e23ff6dc01b40b8bb2bbe4bacf8fee07c566561afd211d1d3097\": container with ID starting with 2b3cde8cadc4e23ff6dc01b40b8bb2bbe4bacf8fee07c566561afd211d1d3097 not found: ID does not exist" containerID="2b3cde8cadc4e23ff6dc01b40b8bb2bbe4bacf8fee07c566561afd211d1d3097"
Nov 02 02:12:38.165634 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.165542    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3cde8cadc4e23ff6dc01b40b8bb2bbe4bacf8fee07c566561afd211d1d3097"} err="failed to get container status \"2b3cde8cadc4e23ff6dc01b40b8bb2bbe4bacf8fee07c566561afd211d1d3097\": rpc error: code = NotFound desc = could not find container \"2b3cde8cadc4e23ff6dc01b40b8bb2bbe4bacf8fee07c566561afd211d1d3097\": container with ID starting with 2b3cde8cadc4e23ff6dc01b40b8bb2bbe4bacf8fee07c566561afd211d1d3097 not found: ID does not exist"
Nov 02 02:12:38.305075 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.304788    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d" podStartSLOduration=2.304772554 podStartE2EDuration="2.304772554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:12:38.304271987 +0000 UTC m=+2405.745308484" watchObservedRunningTime="2026-11-02 02:12:38.304772554 +0000 UTC m=+2405.745809032"
Nov 02 02:12:38.416141 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.416109    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s"]
Nov 02 02:12:38.420103 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.420078    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-092079-5cddb7cb87-6rd9s"]
Nov 02 02:12:38.528611 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.528574    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-092079-9c6765d77-fxqv2"]
Nov 02 02:12:38.532705 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.532679    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-092079-9c6765d77-fxqv2"]
Nov 02 02:12:38.694657 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.694584    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="051907ab-4e37-4298-9291-0df5fa65d735" path="/var/lib/kubelet/pods/051907ab-4e37-4298-9291-0df5fa65d735/volumes"
Nov 02 02:12:38.695024 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:38.695010    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5ae28f8-e723-4bb3-b368-6fb9881d24bf" path="/var/lib/kubelet/pods/c5ae28f8-e723-4bb3-b368-6fb9881d24bf/volumes"
Nov 02 02:12:39.144866 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:39.144830    5875 generic.go:356] "Generic (PLEG): container finished" podID="b2a2a075-44e5-47a4-bcbf-f04d16246a35" containerID="29c493e5452b5dffc7b7bf20fe44fee30d3e8550582be8a75737f874b4e78eb6" exitCode=0
Nov 02 02:12:39.145302 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:39.144909    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d" event={"ID":"b2a2a075-44e5-47a4-bcbf-f04d16246a35","Type":"ContainerDied","Data":"29c493e5452b5dffc7b7bf20fe44fee30d3e8550582be8a75737f874b4e78eb6"}
Nov 02 02:12:40.301286 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:40.301264    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d"
Nov 02 02:12:40.353612 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:40.353583    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2a2a075-44e5-47a4-bcbf-f04d16246a35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2a2a075-44e5-47a4-bcbf-f04d16246a35-maas-api-xks-ca\") pod \"b2a2a075-44e5-47a4-bcbf-f04d16246a35\" (UID: \"b2a2a075-44e5-47a4-bcbf-f04d16246a35\") "
Nov 02 02:12:40.353853 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:40.353832    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2a2a075-44e5-47a4-bcbf-f04d16246a35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2a2a075-44e5-47a4-bcbf-f04d16246a35-maas-api-openshift-ca\") pod \"b2a2a075-44e5-47a4-bcbf-f04d16246a35\" (UID: \"b2a2a075-44e5-47a4-bcbf-f04d16246a35\") "
Nov 02 02:12:40.353917 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:40.353891    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2a2a075-44e5-47a4-bcbf-f04d16246a35-kube-api-access-p9mzt\" (UniqueName: \"kubernetes.io/projected/b2a2a075-44e5-47a4-bcbf-f04d16246a35-kube-api-access-p9mzt\") pod \"b2a2a075-44e5-47a4-bcbf-f04d16246a35\" (UID: \"b2a2a075-44e5-47a4-bcbf-f04d16246a35\") "
Nov 02 02:12:40.354242 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:40.354164    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2a2a075-44e5-47a4-bcbf-f04d16246a35-maas-api-openshift-ca" pod "b2a2a075-44e5-47a4-bcbf-f04d16246a35" (UID: "b2a2a075-44e5-47a4-bcbf-f04d16246a35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:12:40.354470 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:40.354327    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2a2a075-44e5-47a4-bcbf-f04d16246a35-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:40.355759 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:40.355736    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2a2a075-44e5-47a4-bcbf-f04d16246a35-maas-api-xks-ca" pod "b2a2a075-44e5-47a4-bcbf-f04d16246a35" (UID: "b2a2a075-44e5-47a4-bcbf-f04d16246a35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:12:40.355996 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:40.355979    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a2a075-44e5-47a4-bcbf-f04d16246a35-kube-api-access-p9mzt" pod "b2a2a075-44e5-47a4-bcbf-f04d16246a35" (UID: "b2a2a075-44e5-47a4-bcbf-f04d16246a35"). InnerVolumeSpecName "kube-api-access-p9mzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:12:40.455319 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:40.455288    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9mzt\" (UniqueName: \"kubernetes.io/projected/b2a2a075-44e5-47a4-bcbf-f04d16246a35-kube-api-access-p9mzt\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:40.455482 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:40.455326    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2a2a075-44e5-47a4-bcbf-f04d16246a35-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:41.159424 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:41.159389    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d" event={"ID":"b2a2a075-44e5-47a4-bcbf-f04d16246a35","Type":"ContainerDied","Data":"5968c2cde8871fd83d8f9595d2c1fb158ed542b3e7ed8702dd7028d323c2c4ec"}
Nov 02 02:12:41.159424 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:41.159416    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d"
Nov 02 02:12:41.159634 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:41.159437    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5968c2cde8871fd83d8f9595d2c1fb158ed542b3e7ed8702dd7028d323c2c4ec"
Nov 02 02:12:41.323551 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:12:41.323510    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:12:42.840855 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:42.840803    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4" podUID="0e0047bd-4ecb-46b0-80ef-5c40b882dda6" containerName="payload-processing" containerID="cri-o://e212ac312bb499c523a96887e02592bc7e07f0a2e5f56dc9d899e18783bfcfe2" gracePeriod=25
Nov 02 02:12:42.867543 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:42.867513    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4" podUID="0e0047bd-4ecb-46b0-80ef-5c40b882dda6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:12:44.675060 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:44.675020    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" containerID="cri-o://cfcd8a4c3954052dfbcdcb1cf0a5b47e02500d041cd974e36a15cfaa1ed356d8" gracePeriod=45
Nov 02 02:12:44.939553 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:44.939492    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 02:12:45.009314 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.009278    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2378991-0555-45c0-9b55-641a08637893-kube-api-access-7jfvx\" (UniqueName: \"kubernetes.io/projected/c2378991-0555-45c0-9b55-641a08637893-kube-api-access-7jfvx\") pod \"c2378991-0555-45c0-9b55-641a08637893\" (UID: \"c2378991-0555-45c0-9b55-641a08637893\") "
Nov 02 02:12:45.009494 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.009361    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2378991-0555-45c0-9b55-641a08637893-tls-certs\" (UniqueName: \"kubernetes.io/secret/c2378991-0555-45c0-9b55-641a08637893-tls-certs\") pod \"c2378991-0555-45c0-9b55-641a08637893\" (UID: \"c2378991-0555-45c0-9b55-641a08637893\") "
Nov 02 02:12:45.009581 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.009494    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-tmp-dir\") pod \"c2378991-0555-45c0-9b55-641a08637893\" (UID: \"c2378991-0555-45c0-9b55-641a08637893\") "
Nov 02 02:12:45.009581 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.009513    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-home\" (UniqueName: \"kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-home\") pod \"c2378991-0555-45c0-9b55-641a08637893\" (UID: \"c2378991-0555-45c0-9b55-641a08637893\") "
Nov 02 02:12:45.009581 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.009533    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-model-cache\") pod \"c2378991-0555-45c0-9b55-641a08637893\" (UID: \"c2378991-0555-45c0-9b55-641a08637893\") "
Nov 02 02:12:45.009581 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.009570    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-dshm\") pod \"c2378991-0555-45c0-9b55-641a08637893\" (UID: \"c2378991-0555-45c0-9b55-641a08637893\") "
Nov 02 02:12:45.009808 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.009775    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-home" pod "c2378991-0555-45c0-9b55-641a08637893" (UID: "c2378991-0555-45c0-9b55-641a08637893"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:45.009808 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.009788    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-tmp-dir" pod "c2378991-0555-45c0-9b55-641a08637893" (UID: "c2378991-0555-45c0-9b55-641a08637893"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:45.009927 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.009807    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-model-cache" pod "c2378991-0555-45c0-9b55-641a08637893" (UID: "c2378991-0555-45c0-9b55-641a08637893"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:45.010154 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.010097    5875 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-tmp-dir\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:45.010154 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.010132    5875 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-home\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:45.010154 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.010141    5875 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-model-cache\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:45.011793 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.011777    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-dshm" pod "c2378991-0555-45c0-9b55-641a08637893" (UID: "c2378991-0555-45c0-9b55-641a08637893"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:45.011909 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.011798    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2378991-0555-45c0-9b55-641a08637893-tls-certs" pod "c2378991-0555-45c0-9b55-641a08637893" (UID: "c2378991-0555-45c0-9b55-641a08637893"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:12:45.011909 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.011822    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2378991-0555-45c0-9b55-641a08637893-kube-api-access-7jfvx" pod "c2378991-0555-45c0-9b55-641a08637893" (UID: "c2378991-0555-45c0-9b55-641a08637893"). InnerVolumeSpecName "kube-api-access-7jfvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:12:45.112033 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.111990    5875 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2378991-0555-45c0-9b55-641a08637893-dshm\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:45.112193 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.112060    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jfvx\" (UniqueName: \"kubernetes.io/projected/c2378991-0555-45c0-9b55-641a08637893-kube-api-access-7jfvx\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:45.112193 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.112077    5875 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2378991-0555-45c0-9b55-641a08637893-tls-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:45.185338 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.185305    5875 generic.go:356] "Generic (PLEG): container finished" podID="c2378991-0555-45c0-9b55-641a08637893" containerID="cfcd8a4c3954052dfbcdcb1cf0a5b47e02500d041cd974e36a15cfaa1ed356d8" exitCode=0
Nov 02 02:12:45.185539 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.185361    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"
Nov 02 02:12:45.185539 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.185385    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" event={"ID":"c2378991-0555-45c0-9b55-641a08637893","Type":"ContainerDied","Data":"cfcd8a4c3954052dfbcdcb1cf0a5b47e02500d041cd974e36a15cfaa1ed356d8"}
Nov 02 02:12:45.185539 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.185440    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" event={"ID":"c2378991-0555-45c0-9b55-641a08637893","Type":"ContainerDied","Data":"3f9cfbe6e51e8f323aa711e54660f78f0006dd4b41eaf400442ef9b513659efe"}
Nov 02 02:12:45.185539 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.185463    5875 scope.go:122] "RemoveContainer" containerID="cfcd8a4c3954052dfbcdcb1cf0a5b47e02500d041cd974e36a15cfaa1ed356d8"
Nov 02 02:12:45.200255 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.200203    5875 scope.go:122] "RemoveContainer" containerID="cfcd8a4c3954052dfbcdcb1cf0a5b47e02500d041cd974e36a15cfaa1ed356d8"
Nov 02 02:12:45.200599 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:12:45.200578    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfcd8a4c3954052dfbcdcb1cf0a5b47e02500d041cd974e36a15cfaa1ed356d8\": container with ID starting with cfcd8a4c3954052dfbcdcb1cf0a5b47e02500d041cd974e36a15cfaa1ed356d8 not found: ID does not exist" containerID="cfcd8a4c3954052dfbcdcb1cf0a5b47e02500d041cd974e36a15cfaa1ed356d8"
Nov 02 02:12:45.200679 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.200610    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfcd8a4c3954052dfbcdcb1cf0a5b47e02500d041cd974e36a15cfaa1ed356d8"} err="failed to get container status \"cfcd8a4c3954052dfbcdcb1cf0a5b47e02500d041cd974e36a15cfaa1ed356d8\": rpc error: code = NotFound desc = could not find container \"cfcd8a4c3954052dfbcdcb1cf0a5b47e02500d041cd974e36a15cfaa1ed356d8\": container with ID starting with cfcd8a4c3954052dfbcdcb1cf0a5b47e02500d041cd974e36a15cfaa1ed356d8 not found: ID does not exist"
Nov 02 02:12:45.349780 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.349738    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"]
Nov 02 02:12:45.358215 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.358183    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8"]
Nov 02 02:12:45.913800 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:45.913755    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9470a3/distinct-2-w0-9470a3-kserve-76465cb67b-w9xx8" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 02:12:46.695476 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:46.695441    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2378991-0555-45c0-9b55-641a08637893" path="/var/lib/kubelet/pods/c2378991-0555-45c0-9b55-641a08637893/volumes"
Nov 02 02:12:48.277951 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:48.277917    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg"]
Nov 02 02:12:50.934196 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:50.934153    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:12:52.881431 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:52.881375    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4" podUID="0e0047bd-4ecb-46b0-80ef-5c40b882dda6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:12:53.285245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:53.284836    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg" podUID="89e5e146-c748-4d7f-aa22-4a77851f8817" containerName="payload-pre-processing" containerID="cri-o://099057a2dad0e178985208dfc3898b04a499917efd20c80ca09b23343ee1e446" gracePeriod=25
Nov 02 02:12:54.998055 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:12:54.998022    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg" podUID="89e5e146-c748-4d7f-aa22-4a77851f8817" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:13:02.866877 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:13:02.866854    5875 logging.go:55] [core] [Channel #5568 SubChannel #5569]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.44:50052", ServerName: "10.134.0.44:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.44:50052: connect: connection refused"
Nov 02 02:13:03.100734 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:03.100670    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4"
Nov 02 02:13:03.211973 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:03.211944    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e0047bd-4ecb-46b0-80ef-5c40b882dda6-kube-api-access-67rn6\" (UniqueName: \"kubernetes.io/projected/0e0047bd-4ecb-46b0-80ef-5c40b882dda6-kube-api-access-67rn6\") pod \"0e0047bd-4ecb-46b0-80ef-5c40b882dda6\" (UID: \"0e0047bd-4ecb-46b0-80ef-5c40b882dda6\") "
Nov 02 02:13:03.212142 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:03.212056    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e0047bd-4ecb-46b0-80ef-5c40b882dda6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e0047bd-4ecb-46b0-80ef-5c40b882dda6-plugins-config-volume\") pod \"0e0047bd-4ecb-46b0-80ef-5c40b882dda6\" (UID: \"0e0047bd-4ecb-46b0-80ef-5c40b882dda6\") "
Nov 02 02:13:03.212475 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:03.212452    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e0047bd-4ecb-46b0-80ef-5c40b882dda6-plugins-config-volume" pod "0e0047bd-4ecb-46b0-80ef-5c40b882dda6" (UID: "0e0047bd-4ecb-46b0-80ef-5c40b882dda6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:13:03.212639 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:03.212625    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e0047bd-4ecb-46b0-80ef-5c40b882dda6-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:03.214025 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:03.214005    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e0047bd-4ecb-46b0-80ef-5c40b882dda6-kube-api-access-67rn6" pod "0e0047bd-4ecb-46b0-80ef-5c40b882dda6" (UID: "0e0047bd-4ecb-46b0-80ef-5c40b882dda6"). InnerVolumeSpecName "kube-api-access-67rn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:13:03.314125 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:03.314094    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67rn6\" (UniqueName: \"kubernetes.io/projected/0e0047bd-4ecb-46b0-80ef-5c40b882dda6-kube-api-access-67rn6\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:03.328069 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:03.328038    5875 generic.go:356] "Generic (PLEG): container finished" podID="0e0047bd-4ecb-46b0-80ef-5c40b882dda6" containerID="e212ac312bb499c523a96887e02592bc7e07f0a2e5f56dc9d899e18783bfcfe2" exitCode=0
Nov 02 02:13:03.328270 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:03.328098    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4"
Nov 02 02:13:03.328270 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:03.328126    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4" event={"ID":"0e0047bd-4ecb-46b0-80ef-5c40b882dda6","Type":"ContainerDied","Data":"e212ac312bb499c523a96887e02592bc7e07f0a2e5f56dc9d899e18783bfcfe2"}
Nov 02 02:13:03.328270 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:03.328188    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4" event={"ID":"0e0047bd-4ecb-46b0-80ef-5c40b882dda6","Type":"ContainerDied","Data":"96df780073e845e42e7b7092c3f56a5a7e4b1dd1d376bfa880f9ca4e8f55910e"}
Nov 02 02:13:03.328270 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:03.328216    5875 scope.go:122] "RemoveContainer" containerID="e212ac312bb499c523a96887e02592bc7e07f0a2e5f56dc9d899e18783bfcfe2"
Nov 02 02:13:03.338468 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:03.338450    5875 scope.go:122] "RemoveContainer" containerID="e212ac312bb499c523a96887e02592bc7e07f0a2e5f56dc9d899e18783bfcfe2"
Nov 02 02:13:03.338725 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:13:03.338708    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e212ac312bb499c523a96887e02592bc7e07f0a2e5f56dc9d899e18783bfcfe2\": container with ID starting with e212ac312bb499c523a96887e02592bc7e07f0a2e5f56dc9d899e18783bfcfe2 not found: ID does not exist" containerID="e212ac312bb499c523a96887e02592bc7e07f0a2e5f56dc9d899e18783bfcfe2"
Nov 02 02:13:03.338791 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:03.338733    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e212ac312bb499c523a96887e02592bc7e07f0a2e5f56dc9d899e18783bfcfe2"} err="failed to get container status \"e212ac312bb499c523a96887e02592bc7e07f0a2e5f56dc9d899e18783bfcfe2\": rpc error: code = NotFound desc = could not find container \"e212ac312bb499c523a96887e02592bc7e07f0a2e5f56dc9d899e18783bfcfe2\": container with ID starting with e212ac312bb499c523a96887e02592bc7e07f0a2e5f56dc9d899e18783bfcfe2 not found: ID does not exist"
Nov 02 02:13:03.493124 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:03.493086    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4"]
Nov 02 02:13:03.499526 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:03.499501    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4"]
Nov 02 02:13:03.867361 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:03.867270    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-9470a3-dd54cdb9f-sglx4" podUID="0e0047bd-4ecb-46b0-80ef-5c40b882dda6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.44:50052\" within 1s: context deadline exceeded"
Nov 02 02:13:04.693754 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:04.693721    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e0047bd-4ecb-46b0-80ef-5c40b882dda6" path="/var/lib/kubelet/pods/0e0047bd-4ecb-46b0-80ef-5c40b882dda6/volumes"
Nov 02 02:13:04.982716 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:04.982680    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83674a01-9srq4"]
Nov 02 02:13:04.983554 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:04.983532    5875 cpu_manager.go:414] "containerMap: removing container" podUID="c5ae28f8-e723-4bb3-b368-6fb9881d24bf" containerName="payload-processing"
Nov 02 02:13:04.983689 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:04.983556    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5ae28f8-e723-4bb3-b368-6fb9881d24bf" containerName="payload-processing"
Nov 02 02:13:04.983689 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:04.983579    5875 cpu_manager.go:414] "containerMap: removing container" podUID="051907ab-4e37-4298-9291-0df5fa65d735" containerName="payload-pre-processing"
Nov 02 02:13:04.983689 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:04.983587    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="051907ab-4e37-4298-9291-0df5fa65d735" containerName="payload-pre-processing"
Nov 02 02:13:04.983689 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:04.983601    5875 cpu_manager.go:414] "containerMap: removing container" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main"
Nov 02 02:13:04.983689 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:04.983608    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main"
Nov 02 02:13:04.983689 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:04.983663    5875 cpu_manager.go:414] "containerMap: removing container" podUID="b2a2a075-44e5-47a4-bcbf-f04d16246a35" containerName="revoke-keys"
Nov 02 02:13:04.983689 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:04.983670    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a2a075-44e5-47a4-bcbf-f04d16246a35" containerName="revoke-keys"
Nov 02 02:13:04.983689 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:04.983688    5875 cpu_manager.go:414] "containerMap: removing container" podUID="0e0047bd-4ecb-46b0-80ef-5c40b882dda6" containerName="payload-processing"
Nov 02 02:13:04.984111 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:04.983695    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e0047bd-4ecb-46b0-80ef-5c40b882dda6" containerName="payload-processing"
Nov 02 02:13:04.984111 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:04.983845    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e0047bd-4ecb-46b0-80ef-5c40b882dda6" containerName="payload-processing"
Nov 02 02:13:04.984111 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:04.983864    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5ae28f8-e723-4bb3-b368-6fb9881d24bf" containerName="payload-processing"
Nov 02 02:13:04.984111 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:04.983877    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="051907ab-4e37-4298-9291-0df5fa65d735" containerName="payload-pre-processing"
Nov 02 02:13:04.984111 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:04.983888    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2378991-0555-45c0-9b55-641a08637893" containerName="main"
Nov 02 02:13:04.984111 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:04.983905    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a2a075-44e5-47a4-bcbf-f04d16246a35" containerName="revoke-keys"
Nov 02 02:13:04.988498 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:04.988478    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83674a01-9srq4"
Nov 02 02:13:04.998439 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:04.998417    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg" podUID="89e5e146-c748-4d7f-aa22-4a77851f8817" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:13:05.042054 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.041023    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aa3a765-qhw9f"]
Nov 02 02:13:05.045439 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.045423    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa3a765-qhw9f"
Nov 02 02:13:05.087555 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.087530    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:13:05.087733 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.087535    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:13:05.091912 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.091884    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83674a01-9srq4"]
Nov 02 02:13:05.103631 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.103592    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aa3a765-qhw9f"]
Nov 02 02:13:05.134760 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.134724    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e53ad90e-2453-491f-a47f-e9f25ce72b18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83674a01-9srq4\" (UID: \"e53ad90e-2453-491f-a47f-e9f25ce72b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83674a01-9srq4"
Nov 02 02:13:05.135040 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.134805    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e53ad90e-2453-491f-a47f-e9f25ce72b18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83674a01-9srq4\" (UID: \"e53ad90e-2453-491f-a47f-e9f25ce72b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83674a01-9srq4"
Nov 02 02:13:05.135040 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.135024    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk88b\" (UniqueName: \"kubernetes.io/projected/e53ad90e-2453-491f-a47f-e9f25ce72b18-kube-api-access-mk88b\") pod \"maas-api-revoke-sub-83674a01-9srq4\" (UID: \"e53ad90e-2453-491f-a47f-e9f25ce72b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83674a01-9srq4"
Nov 02 02:13:05.236031 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.235954    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92pjs\" (UniqueName: \"kubernetes.io/projected/3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae-kube-api-access-92pjs\") pod \"maas-api-revoke-sub-6aa3a765-qhw9f\" (UID: \"3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa3a765-qhw9f"
Nov 02 02:13:05.236184 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.236088    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk88b\" (UniqueName: \"kubernetes.io/projected/e53ad90e-2453-491f-a47f-e9f25ce72b18-kube-api-access-mk88b\") pod \"maas-api-revoke-sub-83674a01-9srq4\" (UID: \"e53ad90e-2453-491f-a47f-e9f25ce72b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83674a01-9srq4"
Nov 02 02:13:05.236184 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.236126    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6aa3a765-qhw9f\" (UID: \"3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa3a765-qhw9f"
Nov 02 02:13:05.236184 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.236172    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6aa3a765-qhw9f\" (UID: \"3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa3a765-qhw9f"
Nov 02 02:13:05.236335 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.236235    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e53ad90e-2453-491f-a47f-e9f25ce72b18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83674a01-9srq4\" (UID: \"e53ad90e-2453-491f-a47f-e9f25ce72b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83674a01-9srq4"
Nov 02 02:13:05.236410 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.236384    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e53ad90e-2453-491f-a47f-e9f25ce72b18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83674a01-9srq4\" (UID: \"e53ad90e-2453-491f-a47f-e9f25ce72b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83674a01-9srq4"
Nov 02 02:13:05.236896 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.236874    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e53ad90e-2453-491f-a47f-e9f25ce72b18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83674a01-9srq4\" (UID: \"e53ad90e-2453-491f-a47f-e9f25ce72b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83674a01-9srq4"
Nov 02 02:13:05.238832 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.238811    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e53ad90e-2453-491f-a47f-e9f25ce72b18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83674a01-9srq4\" (UID: \"e53ad90e-2453-491f-a47f-e9f25ce72b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83674a01-9srq4"
Nov 02 02:13:05.297378 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.297350    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk88b\" (UniqueName: \"kubernetes.io/projected/e53ad90e-2453-491f-a47f-e9f25ce72b18-kube-api-access-mk88b\") pod \"maas-api-revoke-sub-83674a01-9srq4\" (UID: \"e53ad90e-2453-491f-a47f-e9f25ce72b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83674a01-9srq4"
Nov 02 02:13:05.302009 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.301991    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83674a01-9srq4"
Nov 02 02:13:05.338086 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.338059    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92pjs\" (UniqueName: \"kubernetes.io/projected/3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae-kube-api-access-92pjs\") pod \"maas-api-revoke-sub-6aa3a765-qhw9f\" (UID: \"3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa3a765-qhw9f"
Nov 02 02:13:05.338257 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.338199    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6aa3a765-qhw9f\" (UID: \"3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa3a765-qhw9f"
Nov 02 02:13:05.338751 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.338727    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6aa3a765-qhw9f\" (UID: \"3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa3a765-qhw9f"
Nov 02 02:13:05.339547 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.339520    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6aa3a765-qhw9f\" (UID: \"3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa3a765-qhw9f"
Nov 02 02:13:05.340708 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.340685    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6aa3a765-qhw9f\" (UID: \"3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa3a765-qhw9f"
Nov 02 02:13:05.400734 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.400700    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92pjs\" (UniqueName: \"kubernetes.io/projected/3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae-kube-api-access-92pjs\") pod \"maas-api-revoke-sub-6aa3a765-qhw9f\" (UID: \"3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa3a765-qhw9f"
Nov 02 02:13:05.658601 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.658528    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa3a765-qhw9f"
Nov 02 02:13:05.701288 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:05.701045    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83674a01-9srq4"]
Nov 02 02:13:05.701894 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:13:05.701851    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode53ad90e_2453_491f_a47f_e9f25ce72b18.slice/crio-416c20997604b997b74fab7aac4e6a071af98972cda5f9a28e644982d5de52c8 WatchSource:0}: Error finding container 416c20997604b997b74fab7aac4e6a071af98972cda5f9a28e644982d5de52c8: Status 404 returned error can't find the container with id 416c20997604b997b74fab7aac4e6a071af98972cda5f9a28e644982d5de52c8
Nov 02 02:13:06.067848 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:06.066320    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aa3a765-qhw9f"]
Nov 02 02:13:06.069054 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:13:06.069007    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1cd1ad_cbd0_4d82_beb1_f6dc1769caae.slice/crio-885675f38f7f2cf296d18a2b6d30b2a3a8fddb5b75b82390eed4a601f855d9d1 WatchSource:0}: Error finding container 885675f38f7f2cf296d18a2b6d30b2a3a8fddb5b75b82390eed4a601f855d9d1: Status 404 returned error can't find the container with id 885675f38f7f2cf296d18a2b6d30b2a3a8fddb5b75b82390eed4a601f855d9d1
Nov 02 02:13:06.361438 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:06.361385    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa3a765-qhw9f" event={"ID":"3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae","Type":"ContainerStarted","Data":"885675f38f7f2cf296d18a2b6d30b2a3a8fddb5b75b82390eed4a601f855d9d1"}
Nov 02 02:13:06.363057 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:06.363026    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83674a01-9srq4" event={"ID":"e53ad90e-2453-491f-a47f-e9f25ce72b18","Type":"ContainerStarted","Data":"2f9d7c15014132ae1bb0e5ebba8efcace8e7f23da5ef64833263fbfe3f34f931"}
Nov 02 02:13:06.363201 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:06.363075    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83674a01-9srq4" event={"ID":"e53ad90e-2453-491f-a47f-e9f25ce72b18","Type":"ContainerStarted","Data":"416c20997604b997b74fab7aac4e6a071af98972cda5f9a28e644982d5de52c8"}
Nov 02 02:13:06.479127 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:06.479075    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83674a01-9srq4" podStartSLOduration=2.479060118 podStartE2EDuration="2.479060118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:13:06.47727911 +0000 UTC m=+2433.918315607" watchObservedRunningTime="2026-11-02 02:13:06.479060118 +0000 UTC m=+2433.920096615"
Nov 02 02:13:07.370616 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:07.370534    5875 generic.go:356] "Generic (PLEG): container finished" podID="3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae" containerID="f9d08cdbac86114fae4fbe47da664233f32ff88339710981c7402b57e3ff5b94" exitCode=0
Nov 02 02:13:07.371102 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:07.370611    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa3a765-qhw9f" event={"ID":"3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae","Type":"ContainerDied","Data":"f9d08cdbac86114fae4fbe47da664233f32ff88339710981c7402b57e3ff5b94"}
Nov 02 02:13:07.372295 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:07.372272    5875 generic.go:356] "Generic (PLEG): container finished" podID="e53ad90e-2453-491f-a47f-e9f25ce72b18" containerID="2f9d7c15014132ae1bb0e5ebba8efcace8e7f23da5ef64833263fbfe3f34f931" exitCode=0
Nov 02 02:13:07.372400 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:07.372304    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83674a01-9srq4" event={"ID":"e53ad90e-2453-491f-a47f-e9f25ce72b18","Type":"ContainerDied","Data":"2f9d7c15014132ae1bb0e5ebba8efcace8e7f23da5ef64833263fbfe3f34f931"}
Nov 02 02:13:08.532032 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:08.532006    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83674a01-9srq4"
Nov 02 02:13:08.560398 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:08.560375    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa3a765-qhw9f"
Nov 02 02:13:08.576832 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:08.576811    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e53ad90e-2453-491f-a47f-e9f25ce72b18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e53ad90e-2453-491f-a47f-e9f25ce72b18-maas-api-openshift-ca\") pod \"e53ad90e-2453-491f-a47f-e9f25ce72b18\" (UID: \"e53ad90e-2453-491f-a47f-e9f25ce72b18\") "
Nov 02 02:13:08.576942 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:08.576851    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae-kube-api-access-92pjs\" (UniqueName: \"kubernetes.io/projected/3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae-kube-api-access-92pjs\") pod \"3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae\" (UID: \"3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae\") "
Nov 02 02:13:08.576942 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:08.576900    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e53ad90e-2453-491f-a47f-e9f25ce72b18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e53ad90e-2453-491f-a47f-e9f25ce72b18-maas-api-xks-ca\") pod \"e53ad90e-2453-491f-a47f-e9f25ce72b18\" (UID: \"e53ad90e-2453-491f-a47f-e9f25ce72b18\") "
Nov 02 02:13:08.576942 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:08.576921    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e53ad90e-2453-491f-a47f-e9f25ce72b18-kube-api-access-mk88b\" (UniqueName: \"kubernetes.io/projected/e53ad90e-2453-491f-a47f-e9f25ce72b18-kube-api-access-mk88b\") pod \"e53ad90e-2453-491f-a47f-e9f25ce72b18\" (UID: \"e53ad90e-2453-491f-a47f-e9f25ce72b18\") "
Nov 02 02:13:08.576942 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:08.576936    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae-maas-api-openshift-ca\") pod \"3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae\" (UID: \"3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae\") "
Nov 02 02:13:08.577123 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:08.577027    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae-maas-api-xks-ca\") pod \"3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae\" (UID: \"3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae\") "
Nov 02 02:13:08.577199 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:08.577175    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e53ad90e-2453-491f-a47f-e9f25ce72b18-maas-api-openshift-ca" pod "e53ad90e-2453-491f-a47f-e9f25ce72b18" (UID: "e53ad90e-2453-491f-a47f-e9f25ce72b18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:13:08.577427 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:08.577401    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae-maas-api-openshift-ca" pod "3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae" (UID: "3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:13:08.577522 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:08.577491    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e53ad90e-2453-491f-a47f-e9f25ce72b18-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:08.577522 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:08.577511    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:08.579108 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:08.579085    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae-kube-api-access-92pjs" pod "3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae" (UID: "3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae"). InnerVolumeSpecName "kube-api-access-92pjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:13:08.579301 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:08.579276    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae-maas-api-xks-ca" pod "3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae" (UID: "3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:13:08.579594 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:08.579566    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e53ad90e-2453-491f-a47f-e9f25ce72b18-maas-api-xks-ca" pod "e53ad90e-2453-491f-a47f-e9f25ce72b18" (UID: "e53ad90e-2453-491f-a47f-e9f25ce72b18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:13:08.579594 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:08.579581    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e53ad90e-2453-491f-a47f-e9f25ce72b18-kube-api-access-mk88b" pod "e53ad90e-2453-491f-a47f-e9f25ce72b18" (UID: "e53ad90e-2453-491f-a47f-e9f25ce72b18"). InnerVolumeSpecName "kube-api-access-mk88b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:13:08.678868 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:08.678798    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e53ad90e-2453-491f-a47f-e9f25ce72b18-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:08.678868 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:08.678823    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk88b\" (UniqueName: \"kubernetes.io/projected/e53ad90e-2453-491f-a47f-e9f25ce72b18-kube-api-access-mk88b\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:08.678868 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:08.678833    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:08.678868 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:08.678842    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92pjs\" (UniqueName: \"kubernetes.io/projected/3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae-kube-api-access-92pjs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:09.385342 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:09.385306    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa3a765-qhw9f"
Nov 02 02:13:09.385342 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:09.385329    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa3a765-qhw9f" event={"ID":"3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae","Type":"ContainerDied","Data":"885675f38f7f2cf296d18a2b6d30b2a3a8fddb5b75b82390eed4a601f855d9d1"}
Nov 02 02:13:09.385594 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:09.385381    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="885675f38f7f2cf296d18a2b6d30b2a3a8fddb5b75b82390eed4a601f855d9d1"
Nov 02 02:13:09.386794 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:09.386772    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83674a01-9srq4"
Nov 02 02:13:09.387012 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:09.386790    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83674a01-9srq4" event={"ID":"e53ad90e-2453-491f-a47f-e9f25ce72b18","Type":"ContainerDied","Data":"416c20997604b997b74fab7aac4e6a071af98972cda5f9a28e644982d5de52c8"}
Nov 02 02:13:09.387012 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:09.386826    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="416c20997604b997b74fab7aac4e6a071af98972cda5f9a28e644982d5de52c8"
Nov 02 02:13:11.318981 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:13:11.318897    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:13:13.423568 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:13.423535    5875 generic.go:356] "Generic (PLEG): container finished" podID="89e5e146-c748-4d7f-aa22-4a77851f8817" containerID="099057a2dad0e178985208dfc3898b04a499917efd20c80ca09b23343ee1e446" exitCode=0
Nov 02 02:13:13.423925 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:13.423607    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg" event={"ID":"89e5e146-c748-4d7f-aa22-4a77851f8817","Type":"ContainerDied","Data":"099057a2dad0e178985208dfc3898b04a499917efd20c80ca09b23343ee1e446"}
Nov 02 02:13:13.474339 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:13.474311    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg"
Nov 02 02:13:13.531481 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:13.531452    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89e5e146-c748-4d7f-aa22-4a77851f8817-kube-api-access-7mjzt\" (UniqueName: \"kubernetes.io/projected/89e5e146-c748-4d7f-aa22-4a77851f8817-kube-api-access-7mjzt\") pod \"89e5e146-c748-4d7f-aa22-4a77851f8817\" (UID: \"89e5e146-c748-4d7f-aa22-4a77851f8817\") "
Nov 02 02:13:13.531623 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:13.531497    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89e5e146-c748-4d7f-aa22-4a77851f8817-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89e5e146-c748-4d7f-aa22-4a77851f8817-plugins-config-volume\") pod \"89e5e146-c748-4d7f-aa22-4a77851f8817\" (UID: \"89e5e146-c748-4d7f-aa22-4a77851f8817\") "
Nov 02 02:13:13.531951 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:13.531927    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e5e146-c748-4d7f-aa22-4a77851f8817-plugins-config-volume" pod "89e5e146-c748-4d7f-aa22-4a77851f8817" (UID: "89e5e146-c748-4d7f-aa22-4a77851f8817"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:13:13.533710 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:13.533682    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e5e146-c748-4d7f-aa22-4a77851f8817-kube-api-access-7mjzt" pod "89e5e146-c748-4d7f-aa22-4a77851f8817" (UID: "89e5e146-c748-4d7f-aa22-4a77851f8817"). InnerVolumeSpecName "kube-api-access-7mjzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:13:13.633122 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:13.633045    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89e5e146-c748-4d7f-aa22-4a77851f8817-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:13.633122 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:13.633076    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mjzt\" (UniqueName: \"kubernetes.io/projected/89e5e146-c748-4d7f-aa22-4a77851f8817-kube-api-access-7mjzt\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:14.432096 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.432067    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg"
Nov 02 02:13:14.432517 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.432096    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg" event={"ID":"89e5e146-c748-4d7f-aa22-4a77851f8817","Type":"ContainerDied","Data":"e29679262f94f2a0c56d6359d8ba3beb0110c17be4bbcbcb9e9582259750e592"}
Nov 02 02:13:14.432517 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.432164    5875 scope.go:122] "RemoveContainer" containerID="099057a2dad0e178985208dfc3898b04a499917efd20c80ca09b23343ee1e446"
Nov 02 02:13:14.595334 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.595295    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg"]
Nov 02 02:13:14.644772 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.644739    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-dbe849-84b5f89df5-6gsxg"]
Nov 02 02:13:14.695183 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.695114    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89e5e146-c748-4d7f-aa22-4a77851f8817" path="/var/lib/kubelet/pods/89e5e146-c748-4d7f-aa22-4a77851f8817/volumes"
Nov 02 02:13:14.785308 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.785278    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a02c918e-x6797"]
Nov 02 02:13:14.785796 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.785782    5875 cpu_manager.go:414] "containerMap: removing container" podUID="89e5e146-c748-4d7f-aa22-4a77851f8817" containerName="payload-pre-processing"
Nov 02 02:13:14.785848 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.785797    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89e5e146-c748-4d7f-aa22-4a77851f8817" containerName="payload-pre-processing"
Nov 02 02:13:14.785885 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.785853    5875 cpu_manager.go:414] "containerMap: removing container" podUID="3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae" containerName="revoke-keys"
Nov 02 02:13:14.785885 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.785859    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae" containerName="revoke-keys"
Nov 02 02:13:14.785885 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.785869    5875 cpu_manager.go:414] "containerMap: removing container" podUID="e53ad90e-2453-491f-a47f-e9f25ce72b18" containerName="revoke-keys"
Nov 02 02:13:14.785885 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.785874    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e53ad90e-2453-491f-a47f-e9f25ce72b18" containerName="revoke-keys"
Nov 02 02:13:14.786010 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.785947    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="89e5e146-c748-4d7f-aa22-4a77851f8817" containerName="payload-pre-processing"
Nov 02 02:13:14.786010 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.785963    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae" containerName="revoke-keys"
Nov 02 02:13:14.786010 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.785970    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="e53ad90e-2453-491f-a47f-e9f25ce72b18" containerName="revoke-keys"
Nov 02 02:13:14.790426 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.790411    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02c918e-x6797"
Nov 02 02:13:14.846605 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.846574    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7218bf-2867-4f6d-bdb1-67678e906ea0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a02c918e-x6797\" (UID: \"2e7218bf-2867-4f6d-bdb1-67678e906ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02c918e-x6797"
Nov 02 02:13:14.846766 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.846642    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7218bf-2867-4f6d-bdb1-67678e906ea0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a02c918e-x6797\" (UID: \"2e7218bf-2867-4f6d-bdb1-67678e906ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02c918e-x6797"
Nov 02 02:13:14.846766 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.846715    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcp6s\" (UniqueName: \"kubernetes.io/projected/2e7218bf-2867-4f6d-bdb1-67678e906ea0-kube-api-access-xcp6s\") pod \"maas-api-revoke-sub-a02c918e-x6797\" (UID: \"2e7218bf-2867-4f6d-bdb1-67678e906ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02c918e-x6797"
Nov 02 02:13:14.846880 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.846865    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a02c918e-x6797"]
Nov 02 02:13:14.849346 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.849322    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9635733-b95tr"]
Nov 02 02:13:14.854008 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.853989    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9635733-b95tr"
Nov 02 02:13:14.858985 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.858961    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:13:14.859173 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.858982    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:13:14.912767 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.912727    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9635733-b95tr"]
Nov 02 02:13:14.947664 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.947593    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebb7498-7d43-456b-a98c-8559a858db6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9635733-b95tr\" (UID: \"6ebb7498-7d43-456b-a98c-8559a858db6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9635733-b95tr"
Nov 02 02:13:14.947829 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.947766    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7218bf-2867-4f6d-bdb1-67678e906ea0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a02c918e-x6797\" (UID: \"2e7218bf-2867-4f6d-bdb1-67678e906ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02c918e-x6797"
Nov 02 02:13:14.947829 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.947813    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lxkx\" (UniqueName: \"kubernetes.io/projected/6ebb7498-7d43-456b-a98c-8559a858db6a-kube-api-access-8lxkx\") pod \"maas-api-revoke-sub-c9635733-b95tr\" (UID: \"6ebb7498-7d43-456b-a98c-8559a858db6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9635733-b95tr"
Nov 02 02:13:14.948559 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.948531    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7218bf-2867-4f6d-bdb1-67678e906ea0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a02c918e-x6797\" (UID: \"2e7218bf-2867-4f6d-bdb1-67678e906ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02c918e-x6797"
Nov 02 02:13:14.948837 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.948812    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcp6s\" (UniqueName: \"kubernetes.io/projected/2e7218bf-2867-4f6d-bdb1-67678e906ea0-kube-api-access-xcp6s\") pod \"maas-api-revoke-sub-a02c918e-x6797\" (UID: \"2e7218bf-2867-4f6d-bdb1-67678e906ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02c918e-x6797"
Nov 02 02:13:14.948946 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.948895    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebb7498-7d43-456b-a98c-8559a858db6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9635733-b95tr\" (UID: \"6ebb7498-7d43-456b-a98c-8559a858db6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9635733-b95tr"
Nov 02 02:13:14.949192 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.949157    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7218bf-2867-4f6d-bdb1-67678e906ea0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a02c918e-x6797\" (UID: \"2e7218bf-2867-4f6d-bdb1-67678e906ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02c918e-x6797"
Nov 02 02:13:14.952359 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:14.952331    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7218bf-2867-4f6d-bdb1-67678e906ea0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a02c918e-x6797\" (UID: \"2e7218bf-2867-4f6d-bdb1-67678e906ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02c918e-x6797"
Nov 02 02:13:15.010858 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:15.010826    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcp6s\" (UniqueName: \"kubernetes.io/projected/2e7218bf-2867-4f6d-bdb1-67678e906ea0-kube-api-access-xcp6s\") pod \"maas-api-revoke-sub-a02c918e-x6797\" (UID: \"2e7218bf-2867-4f6d-bdb1-67678e906ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02c918e-x6797"
Nov 02 02:13:15.049854 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:15.049827    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lxkx\" (UniqueName: \"kubernetes.io/projected/6ebb7498-7d43-456b-a98c-8559a858db6a-kube-api-access-8lxkx\") pod \"maas-api-revoke-sub-c9635733-b95tr\" (UID: \"6ebb7498-7d43-456b-a98c-8559a858db6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9635733-b95tr"
Nov 02 02:13:15.049994 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:15.049901    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebb7498-7d43-456b-a98c-8559a858db6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9635733-b95tr\" (UID: \"6ebb7498-7d43-456b-a98c-8559a858db6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9635733-b95tr"
Nov 02 02:13:15.049994 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:15.049926    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebb7498-7d43-456b-a98c-8559a858db6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9635733-b95tr\" (UID: \"6ebb7498-7d43-456b-a98c-8559a858db6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9635733-b95tr"
Nov 02 02:13:15.050567 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:15.050542    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebb7498-7d43-456b-a98c-8559a858db6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9635733-b95tr\" (UID: \"6ebb7498-7d43-456b-a98c-8559a858db6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9635733-b95tr"
Nov 02 02:13:15.052352 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:15.052332    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebb7498-7d43-456b-a98c-8559a858db6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9635733-b95tr\" (UID: \"6ebb7498-7d43-456b-a98c-8559a858db6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9635733-b95tr"
Nov 02 02:13:15.103618 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:15.103594    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02c918e-x6797"
Nov 02 02:13:15.144835 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:15.126602    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lxkx\" (UniqueName: \"kubernetes.io/projected/6ebb7498-7d43-456b-a98c-8559a858db6a-kube-api-access-8lxkx\") pod \"maas-api-revoke-sub-c9635733-b95tr\" (UID: \"6ebb7498-7d43-456b-a98c-8559a858db6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9635733-b95tr"
Nov 02 02:13:15.170593 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:15.170564    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9635733-b95tr"
Nov 02 02:13:15.525656 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:15.525624    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a02c918e-x6797"]
Nov 02 02:13:15.527676 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:13:15.527650    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7218bf_2867_4f6d_bdb1_67678e906ea0.slice/crio-05bbaeeac926c3fa99a3aa8948e8ae717814aa608680cef5d6d6d85ec1860cca WatchSource:0}: Error finding container 05bbaeeac926c3fa99a3aa8948e8ae717814aa608680cef5d6d6d85ec1860cca: Status 404 returned error can't find the container with id 05bbaeeac926c3fa99a3aa8948e8ae717814aa608680cef5d6d6d85ec1860cca
Nov 02 02:13:15.623312 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:15.623284    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9635733-b95tr"]
Nov 02 02:13:15.624804 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:13:15.624784    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebb7498_7d43_456b_a98c_8559a858db6a.slice/crio-a78c3f0d77ce27ba9092f7f56802ee83747dad3fb64eb70cb095058b631f0312 WatchSource:0}: Error finding container a78c3f0d77ce27ba9092f7f56802ee83747dad3fb64eb70cb095058b631f0312: Status 404 returned error can't find the container with id a78c3f0d77ce27ba9092f7f56802ee83747dad3fb64eb70cb095058b631f0312
Nov 02 02:13:16.485524 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:16.485483    5875 generic.go:356] "Generic (PLEG): container finished" podID="2e7218bf-2867-4f6d-bdb1-67678e906ea0" containerID="bdf942a19a3ec22f63d9946309f8408715eddf000e7ad2a71887f0f2ee1f65b5" exitCode=0
Nov 02 02:13:16.485733 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:16.485568    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02c918e-x6797" event={"ID":"2e7218bf-2867-4f6d-bdb1-67678e906ea0","Type":"ContainerDied","Data":"bdf942a19a3ec22f63d9946309f8408715eddf000e7ad2a71887f0f2ee1f65b5"}
Nov 02 02:13:16.485733 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:16.485620    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02c918e-x6797" event={"ID":"2e7218bf-2867-4f6d-bdb1-67678e906ea0","Type":"ContainerStarted","Data":"05bbaeeac926c3fa99a3aa8948e8ae717814aa608680cef5d6d6d85ec1860cca"}
Nov 02 02:13:16.487331 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:16.487308    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9635733-b95tr" event={"ID":"6ebb7498-7d43-456b-a98c-8559a858db6a","Type":"ContainerStarted","Data":"d4243afb4ea6465f087a7f557ccd070c080324601fd74ce394da4c987dcb7835"}
Nov 02 02:13:16.487430 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:16.487351    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9635733-b95tr" event={"ID":"6ebb7498-7d43-456b-a98c-8559a858db6a","Type":"ContainerStarted","Data":"a78c3f0d77ce27ba9092f7f56802ee83747dad3fb64eb70cb095058b631f0312"}
Nov 02 02:13:16.717077 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:16.717028    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9635733-b95tr" podStartSLOduration=2.717016499 podStartE2EDuration="2.717016499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:13:16.714275438 +0000 UTC m=+2444.155311936" watchObservedRunningTime="2026-11-02 02:13:16.717016499 +0000 UTC m=+2444.158052996"
Nov 02 02:13:17.197325 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:17.197244    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092079-nzrk4"]
Nov 02 02:13:17.201918 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:17.201888    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-092079-nzrk4"]
Nov 02 02:13:17.361800 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:17.361767    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:13:17.495117 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:17.495084    5875 generic.go:356] "Generic (PLEG): container finished" podID="6ebb7498-7d43-456b-a98c-8559a858db6a" containerID="d4243afb4ea6465f087a7f557ccd070c080324601fd74ce394da4c987dcb7835" exitCode=0
Nov 02 02:13:17.495321 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:17.495150    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9635733-b95tr" event={"ID":"6ebb7498-7d43-456b-a98c-8559a858db6a","Type":"ContainerDied","Data":"d4243afb4ea6465f087a7f557ccd070c080324601fd74ce394da4c987dcb7835"}
Nov 02 02:13:17.653818 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:17.653785    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02c918e-x6797"
Nov 02 02:13:17.805682 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:17.792966    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e7218bf-2867-4f6d-bdb1-67678e906ea0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7218bf-2867-4f6d-bdb1-67678e906ea0-maas-api-xks-ca\") pod \"2e7218bf-2867-4f6d-bdb1-67678e906ea0\" (UID: \"2e7218bf-2867-4f6d-bdb1-67678e906ea0\") "
Nov 02 02:13:17.805682 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:17.793047    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e7218bf-2867-4f6d-bdb1-67678e906ea0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7218bf-2867-4f6d-bdb1-67678e906ea0-maas-api-openshift-ca\") pod \"2e7218bf-2867-4f6d-bdb1-67678e906ea0\" (UID: \"2e7218bf-2867-4f6d-bdb1-67678e906ea0\") "
Nov 02 02:13:17.805682 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:17.793270    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e7218bf-2867-4f6d-bdb1-67678e906ea0-kube-api-access-xcp6s\" (UniqueName: \"kubernetes.io/projected/2e7218bf-2867-4f6d-bdb1-67678e906ea0-kube-api-access-xcp6s\") pod \"2e7218bf-2867-4f6d-bdb1-67678e906ea0\" (UID: \"2e7218bf-2867-4f6d-bdb1-67678e906ea0\") "
Nov 02 02:13:17.805682 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:17.797099    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e7218bf-2867-4f6d-bdb1-67678e906ea0-maas-api-openshift-ca" pod "2e7218bf-2867-4f6d-bdb1-67678e906ea0" (UID: "2e7218bf-2867-4f6d-bdb1-67678e906ea0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:13:17.805682 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:17.797777    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e7218bf-2867-4f6d-bdb1-67678e906ea0-maas-api-xks-ca" pod "2e7218bf-2867-4f6d-bdb1-67678e906ea0" (UID: "2e7218bf-2867-4f6d-bdb1-67678e906ea0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:13:17.806949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:17.806907    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7218bf-2867-4f6d-bdb1-67678e906ea0-kube-api-access-xcp6s" pod "2e7218bf-2867-4f6d-bdb1-67678e906ea0" (UID: "2e7218bf-2867-4f6d-bdb1-67678e906ea0"). InnerVolumeSpecName "kube-api-access-xcp6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:13:17.907245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:17.907092    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcp6s\" (UniqueName: \"kubernetes.io/projected/2e7218bf-2867-4f6d-bdb1-67678e906ea0-kube-api-access-xcp6s\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:17.907245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:17.907131    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e7218bf-2867-4f6d-bdb1-67678e906ea0-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:17.907245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:17.907148    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e7218bf-2867-4f6d-bdb1-67678e906ea0-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:18.502200 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:18.502168    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02c918e-x6797"
Nov 02 02:13:18.502397 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:18.502202    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02c918e-x6797" event={"ID":"2e7218bf-2867-4f6d-bdb1-67678e906ea0","Type":"ContainerDied","Data":"05bbaeeac926c3fa99a3aa8948e8ae717814aa608680cef5d6d6d85ec1860cca"}
Nov 02 02:13:18.502397 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:18.502266    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05bbaeeac926c3fa99a3aa8948e8ae717814aa608680cef5d6d6d85ec1860cca"
Nov 02 02:13:18.664634 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:18.664602    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9635733-b95tr"
Nov 02 02:13:18.694559 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:18.694529    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93c193d8-2d88-4ee4-8682-c654362e27e1" path="/var/lib/kubelet/pods/93c193d8-2d88-4ee4-8682-c654362e27e1/volumes"
Nov 02 02:13:18.716233 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:18.716187    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ebb7498-7d43-456b-a98c-8559a858db6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebb7498-7d43-456b-a98c-8559a858db6a-maas-api-xks-ca\") pod \"6ebb7498-7d43-456b-a98c-8559a858db6a\" (UID: \"6ebb7498-7d43-456b-a98c-8559a858db6a\") "
Nov 02 02:13:18.716397 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:18.716378    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ebb7498-7d43-456b-a98c-8559a858db6a-kube-api-access-8lxkx\" (UniqueName: \"kubernetes.io/projected/6ebb7498-7d43-456b-a98c-8559a858db6a-kube-api-access-8lxkx\") pod \"6ebb7498-7d43-456b-a98c-8559a858db6a\" (UID: \"6ebb7498-7d43-456b-a98c-8559a858db6a\") "
Nov 02 02:13:18.716468 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:18.716449    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ebb7498-7d43-456b-a98c-8559a858db6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebb7498-7d43-456b-a98c-8559a858db6a-maas-api-openshift-ca\") pod \"6ebb7498-7d43-456b-a98c-8559a858db6a\" (UID: \"6ebb7498-7d43-456b-a98c-8559a858db6a\") "
Nov 02 02:13:18.716873 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:18.716802    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ebb7498-7d43-456b-a98c-8559a858db6a-maas-api-openshift-ca" pod "6ebb7498-7d43-456b-a98c-8559a858db6a" (UID: "6ebb7498-7d43-456b-a98c-8559a858db6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:13:18.718354 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:18.718332    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ebb7498-7d43-456b-a98c-8559a858db6a-maas-api-xks-ca" pod "6ebb7498-7d43-456b-a98c-8559a858db6a" (UID: "6ebb7498-7d43-456b-a98c-8559a858db6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:13:18.718514 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:18.718494    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebb7498-7d43-456b-a98c-8559a858db6a-kube-api-access-8lxkx" pod "6ebb7498-7d43-456b-a98c-8559a858db6a" (UID: "6ebb7498-7d43-456b-a98c-8559a858db6a"). InnerVolumeSpecName "kube-api-access-8lxkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:13:18.818277 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:18.818160    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ebb7498-7d43-456b-a98c-8559a858db6a-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:18.818277 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:18.818201    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ebb7498-7d43-456b-a98c-8559a858db6a-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:18.818277 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:18.818234    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lxkx\" (UniqueName: \"kubernetes.io/projected/6ebb7498-7d43-456b-a98c-8559a858db6a-kube-api-access-8lxkx\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:19.510372 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:19.510338    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9635733-b95tr"
Nov 02 02:13:19.510372 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:19.510345    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9635733-b95tr" event={"ID":"6ebb7498-7d43-456b-a98c-8559a858db6a","Type":"ContainerDied","Data":"a78c3f0d77ce27ba9092f7f56802ee83747dad3fb64eb70cb095058b631f0312"}
Nov 02 02:13:19.510637 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:19.510435    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a78c3f0d77ce27ba9092f7f56802ee83747dad3fb64eb70cb095058b631f0312"
Nov 02 02:13:24.479940 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:24.479904    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"]
Nov 02 02:13:24.480389 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:24.480102    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt" podUID="d12fc012-f64e-4504-85dc-0bbabf28cb68" containerName="istio-proxy" containerID="cri-o://60b0da0877514aad2ece5473c0381522077f7dbe303b48dc6dd73a949b6e7ba6" gracePeriod=30
Nov 02 02:13:25.400546 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:25.400515    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76837e-6vdrl"]
Nov 02 02:13:25.401084 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:25.401070    5875 cpu_manager.go:414] "containerMap: removing container" podUID="6ebb7498-7d43-456b-a98c-8559a858db6a" containerName="revoke-keys"
Nov 02 02:13:25.401134 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:25.401086    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ebb7498-7d43-456b-a98c-8559a858db6a" containerName="revoke-keys"
Nov 02 02:13:25.401134 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:25.401133    5875 cpu_manager.go:414] "containerMap: removing container" podUID="2e7218bf-2867-4f6d-bdb1-67678e906ea0" containerName="revoke-keys"
Nov 02 02:13:25.401201 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:25.401138    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e7218bf-2867-4f6d-bdb1-67678e906ea0" containerName="revoke-keys"
Nov 02 02:13:25.401273 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:25.401214    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e7218bf-2867-4f6d-bdb1-67678e906ea0" containerName="revoke-keys"
Nov 02 02:13:25.401273 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:25.401248    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ebb7498-7d43-456b-a98c-8559a858db6a" containerName="revoke-keys"
Nov 02 02:13:25.404667 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:25.404649    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76837e-6vdrl"
Nov 02 02:13:25.459361 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:25.459317    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:13:25.462490 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:25.462459    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76837e-6vdrl"]
Nov 02 02:13:25.506071 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:25.506041    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8800230-a8a9-4750-9cf1-8921946a48a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-76837e-6vdrl\" (UID: \"c8800230-a8a9-4750-9cf1-8921946a48a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76837e-6vdrl"
Nov 02 02:13:25.506426 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:25.506178    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8800230-a8a9-4750-9cf1-8921946a48a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-76837e-6vdrl\" (UID: \"c8800230-a8a9-4750-9cf1-8921946a48a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76837e-6vdrl"
Nov 02 02:13:25.506426 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:25.506251    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nbwv\" (UniqueName: \"kubernetes.io/projected/c8800230-a8a9-4750-9cf1-8921946a48a6-kube-api-access-4nbwv\") pod \"maas-api-revoke-keys-e2e-api-b-76837e-6vdrl\" (UID: \"c8800230-a8a9-4750-9cf1-8921946a48a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76837e-6vdrl"
Nov 02 02:13:25.508508 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:25.508491    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:13:25.607774 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:25.607743    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8800230-a8a9-4750-9cf1-8921946a48a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-76837e-6vdrl\" (UID: \"c8800230-a8a9-4750-9cf1-8921946a48a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76837e-6vdrl"
Nov 02 02:13:25.607957 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:25.607837    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8800230-a8a9-4750-9cf1-8921946a48a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-76837e-6vdrl\" (UID: \"c8800230-a8a9-4750-9cf1-8921946a48a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76837e-6vdrl"
Nov 02 02:13:25.607957 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:25.607891    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nbwv\" (UniqueName: \"kubernetes.io/projected/c8800230-a8a9-4750-9cf1-8921946a48a6-kube-api-access-4nbwv\") pod \"maas-api-revoke-keys-e2e-api-b-76837e-6vdrl\" (UID: \"c8800230-a8a9-4750-9cf1-8921946a48a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76837e-6vdrl"
Nov 02 02:13:25.608468 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:25.608445    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8800230-a8a9-4750-9cf1-8921946a48a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-76837e-6vdrl\" (UID: \"c8800230-a8a9-4750-9cf1-8921946a48a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76837e-6vdrl"
Nov 02 02:13:25.610340 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:25.610316    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8800230-a8a9-4750-9cf1-8921946a48a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-76837e-6vdrl\" (UID: \"c8800230-a8a9-4750-9cf1-8921946a48a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76837e-6vdrl"
Nov 02 02:13:25.668313 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:25.668256    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nbwv\" (UniqueName: \"kubernetes.io/projected/c8800230-a8a9-4750-9cf1-8921946a48a6-kube-api-access-4nbwv\") pod \"maas-api-revoke-keys-e2e-api-b-76837e-6vdrl\" (UID: \"c8800230-a8a9-4750-9cf1-8921946a48a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76837e-6vdrl"
Nov 02 02:13:25.717730 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:25.717708    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76837e-6vdrl"
Nov 02 02:13:26.091547 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:26.091521    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76837e-6vdrl"]
Nov 02 02:13:26.093533 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:13:26.093508    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8800230_a8a9_4750_9cf1_8921946a48a6.slice/crio-ad6852e483cc7d155d385ae058d89b5f497878098b11e5658a71db200c605f93 WatchSource:0}: Error finding container ad6852e483cc7d155d385ae058d89b5f497878098b11e5658a71db200c605f93: Status 404 returned error can't find the container with id ad6852e483cc7d155d385ae058d89b5f497878098b11e5658a71db200c605f93
Nov 02 02:13:26.566012 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:26.565973    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76837e-6vdrl" event={"ID":"c8800230-a8a9-4750-9cf1-8921946a48a6","Type":"ContainerStarted","Data":"f714ac9c3b89517edd66384dfe16095fd6bd550983e1c49ba6748d8976ea3cbe"}
Nov 02 02:13:26.566526 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:26.566026    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76837e-6vdrl" event={"ID":"c8800230-a8a9-4750-9cf1-8921946a48a6","Type":"ContainerStarted","Data":"ad6852e483cc7d155d385ae058d89b5f497878098b11e5658a71db200c605f93"}
Nov 02 02:13:26.681364 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:26.681252    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76837e-6vdrl" podStartSLOduration=1.6812110630000001 podStartE2EDuration="1.681211063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:13:26.680314774 +0000 UTC m=+2454.121351273" watchObservedRunningTime="2026-11-02 02:13:26.681211063 +0000 UTC m=+2454.122247560"
Nov 02 02:13:27.574106 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:27.574067    5875 generic.go:356] "Generic (PLEG): container finished" podID="c8800230-a8a9-4750-9cf1-8921946a48a6" containerID="f714ac9c3b89517edd66384dfe16095fd6bd550983e1c49ba6748d8976ea3cbe" exitCode=0
Nov 02 02:13:27.574518 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:27.574152    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76837e-6vdrl" event={"ID":"c8800230-a8a9-4750-9cf1-8921946a48a6","Type":"ContainerDied","Data":"f714ac9c3b89517edd66384dfe16095fd6bd550983e1c49ba6748d8976ea3cbe"}
Nov 02 02:13:28.719783 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:28.719762    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76837e-6vdrl"
Nov 02 02:13:28.844382 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:28.844305    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8800230-a8a9-4750-9cf1-8921946a48a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8800230-a8a9-4750-9cf1-8921946a48a6-maas-api-openshift-ca\") pod \"c8800230-a8a9-4750-9cf1-8921946a48a6\" (UID: \"c8800230-a8a9-4750-9cf1-8921946a48a6\") "
Nov 02 02:13:28.844574 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:28.844383    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8800230-a8a9-4750-9cf1-8921946a48a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8800230-a8a9-4750-9cf1-8921946a48a6-maas-api-xks-ca\") pod \"c8800230-a8a9-4750-9cf1-8921946a48a6\" (UID: \"c8800230-a8a9-4750-9cf1-8921946a48a6\") "
Nov 02 02:13:28.844574 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:28.844507    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8800230-a8a9-4750-9cf1-8921946a48a6-kube-api-access-4nbwv\" (UniqueName: \"kubernetes.io/projected/c8800230-a8a9-4750-9cf1-8921946a48a6-kube-api-access-4nbwv\") pod \"c8800230-a8a9-4750-9cf1-8921946a48a6\" (UID: \"c8800230-a8a9-4750-9cf1-8921946a48a6\") "
Nov 02 02:13:28.844762 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:28.844732    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8800230-a8a9-4750-9cf1-8921946a48a6-maas-api-openshift-ca" pod "c8800230-a8a9-4750-9cf1-8921946a48a6" (UID: "c8800230-a8a9-4750-9cf1-8921946a48a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:13:28.845018 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:28.844993    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8800230-a8a9-4750-9cf1-8921946a48a6-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:28.846742 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:28.846714    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8800230-a8a9-4750-9cf1-8921946a48a6-maas-api-xks-ca" pod "c8800230-a8a9-4750-9cf1-8921946a48a6" (UID: "c8800230-a8a9-4750-9cf1-8921946a48a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:13:28.847138 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:28.847116    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8800230-a8a9-4750-9cf1-8921946a48a6-kube-api-access-4nbwv" pod "c8800230-a8a9-4750-9cf1-8921946a48a6" (UID: "c8800230-a8a9-4750-9cf1-8921946a48a6"). InnerVolumeSpecName "kube-api-access-4nbwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:13:28.946630 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:28.946599    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nbwv\" (UniqueName: \"kubernetes.io/projected/c8800230-a8a9-4750-9cf1-8921946a48a6-kube-api-access-4nbwv\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:28.946630 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:28.946625    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8800230-a8a9-4750-9cf1-8921946a48a6-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:29.588975 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.588948    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76837e-6vdrl"
Nov 02 02:13:29.588975 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.588949    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76837e-6vdrl" event={"ID":"c8800230-a8a9-4750-9cf1-8921946a48a6","Type":"ContainerDied","Data":"ad6852e483cc7d155d385ae058d89b5f497878098b11e5658a71db200c605f93"}
Nov 02 02:13:29.589174 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.589015    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad6852e483cc7d155d385ae058d89b5f497878098b11e5658a71db200c605f93"
Nov 02 02:13:29.744590 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.744530    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:13:29.855189 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.855161    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d12fc012-f64e-4504-85dc-0bbabf28cb68-kube-api-access-s6dl9\" (UniqueName: \"kubernetes.io/projected/d12fc012-f64e-4504-85dc-0bbabf28cb68-kube-api-access-s6dl9\") pod \"d12fc012-f64e-4504-85dc-0bbabf28cb68\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") "
Nov 02 02:13:29.855394 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.855206    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-workload-socket\") pod \"d12fc012-f64e-4504-85dc-0bbabf28cb68\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") "
Nov 02 02:13:29.855394 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.855325    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-credential-socket\") pod \"d12fc012-f64e-4504-85dc-0bbabf28cb68\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") "
Nov 02 02:13:29.855523 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.855392    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-workload-certs\") pod \"d12fc012-f64e-4504-85dc-0bbabf28cb68\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") "
Nov 02 02:13:29.855523 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.855441    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-data\") pod \"d12fc012-f64e-4504-85dc-0bbabf28cb68\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") "
Nov 02 02:13:29.855523 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.855492    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-token\" (UniqueName: \"kubernetes.io/projected/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-token\") pod \"d12fc012-f64e-4504-85dc-0bbabf28cb68\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") "
Nov 02 02:13:29.855523 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.855519    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-podinfo\") pod \"d12fc012-f64e-4504-85dc-0bbabf28cb68\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") "
Nov 02 02:13:29.855747 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.855556    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-envoy\") pod \"d12fc012-f64e-4504-85dc-0bbabf28cb68\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") "
Nov 02 02:13:29.855747 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.855585    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d12fc012-f64e-4504-85dc-0bbabf28cb68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d12fc012-f64e-4504-85dc-0bbabf28cb68-istiod-ca-cert\") pod \"d12fc012-f64e-4504-85dc-0bbabf28cb68\" (UID: \"d12fc012-f64e-4504-85dc-0bbabf28cb68\") "
Nov 02 02:13:29.855747 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.855589    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-workload-socket" pod "d12fc012-f64e-4504-85dc-0bbabf28cb68" (UID: "d12fc012-f64e-4504-85dc-0bbabf28cb68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:13:29.855747 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.855604    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-credential-socket" pod "d12fc012-f64e-4504-85dc-0bbabf28cb68" (UID: "d12fc012-f64e-4504-85dc-0bbabf28cb68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:13:29.855747 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.855622    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-workload-certs" pod "d12fc012-f64e-4504-85dc-0bbabf28cb68" (UID: "d12fc012-f64e-4504-85dc-0bbabf28cb68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:13:29.855747 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.855727    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-data" pod "d12fc012-f64e-4504-85dc-0bbabf28cb68" (UID: "d12fc012-f64e-4504-85dc-0bbabf28cb68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:13:29.856249 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.855938    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12fc012-f64e-4504-85dc-0bbabf28cb68-istiod-ca-cert" pod "d12fc012-f64e-4504-85dc-0bbabf28cb68" (UID: "d12fc012-f64e-4504-85dc-0bbabf28cb68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:13:29.856249 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.856038    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:29.856249 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.856060    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:29.856249 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.856074    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:29.856249 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.856088    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d12fc012-f64e-4504-85dc-0bbabf28cb68-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:29.856249 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.856102    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:29.857666 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.857641    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-podinfo" pod "d12fc012-f64e-4504-85dc-0bbabf28cb68" (UID: "d12fc012-f64e-4504-85dc-0bbabf28cb68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:13:29.857814 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.857773    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-envoy" pod "d12fc012-f64e-4504-85dc-0bbabf28cb68" (UID: "d12fc012-f64e-4504-85dc-0bbabf28cb68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:13:29.857814 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.857795    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12fc012-f64e-4504-85dc-0bbabf28cb68-kube-api-access-s6dl9" pod "d12fc012-f64e-4504-85dc-0bbabf28cb68" (UID: "d12fc012-f64e-4504-85dc-0bbabf28cb68"). InnerVolumeSpecName "kube-api-access-s6dl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:13:29.857950 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.857893    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-token" pod "d12fc012-f64e-4504-85dc-0bbabf28cb68" (UID: "d12fc012-f64e-4504-85dc-0bbabf28cb68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:13:29.957359 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.957330    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6dl9\" (UniqueName: \"kubernetes.io/projected/d12fc012-f64e-4504-85dc-0bbabf28cb68-kube-api-access-s6dl9\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:29.957359 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.957358    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:29.957559 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.957367    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:29.957559 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:29.957376    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d12fc012-f64e-4504-85dc-0bbabf28cb68-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:30.596682 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:30.596648    5875 generic.go:356] "Generic (PLEG): container finished" podID="d12fc012-f64e-4504-85dc-0bbabf28cb68" containerID="60b0da0877514aad2ece5473c0381522077f7dbe303b48dc6dd73a949b6e7ba6" exitCode=0
Nov 02 02:13:30.596890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:30.596717    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"
Nov 02 02:13:30.596890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:30.596734    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt" event={"ID":"d12fc012-f64e-4504-85dc-0bbabf28cb68","Type":"ContainerDied","Data":"60b0da0877514aad2ece5473c0381522077f7dbe303b48dc6dd73a949b6e7ba6"}
Nov 02 02:13:30.596890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:30.596820    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt" event={"ID":"d12fc012-f64e-4504-85dc-0bbabf28cb68","Type":"ContainerDied","Data":"422fd9608f19a14527800d19ebeb871b74aa2b0de3c8a57034e3c0377160620d"}
Nov 02 02:13:30.596890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:30.596843    5875 scope.go:122] "RemoveContainer" containerID="60b0da0877514aad2ece5473c0381522077f7dbe303b48dc6dd73a949b6e7ba6"
Nov 02 02:13:30.607369 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:30.607354    5875 scope.go:122] "RemoveContainer" containerID="9404db203db3eb9bf9266ed3eb94df6ab4e6a820631fcedd4d864dffeaeb4ff4"
Nov 02 02:13:30.616202 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:30.616176    5875 scope.go:122] "RemoveContainer" containerID="60b0da0877514aad2ece5473c0381522077f7dbe303b48dc6dd73a949b6e7ba6"
Nov 02 02:13:30.616532 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:13:30.616513    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60b0da0877514aad2ece5473c0381522077f7dbe303b48dc6dd73a949b6e7ba6\": container with ID starting with 60b0da0877514aad2ece5473c0381522077f7dbe303b48dc6dd73a949b6e7ba6 not found: ID does not exist" containerID="60b0da0877514aad2ece5473c0381522077f7dbe303b48dc6dd73a949b6e7ba6"
Nov 02 02:13:30.616597 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:30.616541    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60b0da0877514aad2ece5473c0381522077f7dbe303b48dc6dd73a949b6e7ba6"} err="failed to get container status \"60b0da0877514aad2ece5473c0381522077f7dbe303b48dc6dd73a949b6e7ba6\": rpc error: code = NotFound desc = could not find container \"60b0da0877514aad2ece5473c0381522077f7dbe303b48dc6dd73a949b6e7ba6\": container with ID starting with 60b0da0877514aad2ece5473c0381522077f7dbe303b48dc6dd73a949b6e7ba6 not found: ID does not exist"
Nov 02 02:13:30.616597 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:30.616559    5875 scope.go:122] "RemoveContainer" containerID="9404db203db3eb9bf9266ed3eb94df6ab4e6a820631fcedd4d864dffeaeb4ff4"
Nov 02 02:13:30.616818 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:13:30.616800    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9404db203db3eb9bf9266ed3eb94df6ab4e6a820631fcedd4d864dffeaeb4ff4\": container with ID starting with 9404db203db3eb9bf9266ed3eb94df6ab4e6a820631fcedd4d864dffeaeb4ff4 not found: ID does not exist" containerID="9404db203db3eb9bf9266ed3eb94df6ab4e6a820631fcedd4d864dffeaeb4ff4"
Nov 02 02:13:30.616864 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:30.616825    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9404db203db3eb9bf9266ed3eb94df6ab4e6a820631fcedd4d864dffeaeb4ff4"} err="failed to get container status \"9404db203db3eb9bf9266ed3eb94df6ab4e6a820631fcedd4d864dffeaeb4ff4\": rpc error: code = NotFound desc = could not find container \"9404db203db3eb9bf9266ed3eb94df6ab4e6a820631fcedd4d864dffeaeb4ff4\": container with ID starting with 9404db203db3eb9bf9266ed3eb94df6ab4e6a820631fcedd4d864dffeaeb4ff4 not found: ID does not exist"
Nov 02 02:13:30.810896 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:30.810861    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"]
Nov 02 02:13:30.860145 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:30.860080    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-092079-openshift-default-59469cb5ff-npkzt"]
Nov 02 02:13:32.694271 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:32.694243    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d12fc012-f64e-4504-85dc-0bbabf28cb68" path="/var/lib/kubelet/pods/d12fc012-f64e-4504-85dc-0bbabf28cb68/volumes"
Nov 02 02:13:36.865995 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:36.865960    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8"]
Nov 02 02:13:36.974925 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:36.974886    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp"]
Nov 02 02:13:41.317944 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:13:41.317905    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:13:41.873473 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:41.873402    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8" podUID="11cd3f02-9989-4e40-be11-f7e7b72f9275" containerName="payload-processing" containerID="cri-o://d9b64da82181e4aa17563fd09b03c46a6a2788c06954e0ae8955eb15e866caee" gracePeriod=25
Nov 02 02:13:41.976473 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:41.976413    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp" podUID="2584d15c-d3a3-4392-bb37-5b2c650b7cf7" containerName="payload-pre-processing" containerID="cri-o://06c7a2f5ca4e3fdd3f6ead4ee54547d54881443ca0c85d542df36f68e77d867d" gracePeriod=25
Nov 02 02:13:42.271879 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.271853    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8"
Nov 02 02:13:42.275250 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.275202    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp"
Nov 02 02:13:42.401046 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.400974    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11cd3f02-9989-4e40-be11-f7e7b72f9275-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11cd3f02-9989-4e40-be11-f7e7b72f9275-plugins-config-volume\") pod \"11cd3f02-9989-4e40-be11-f7e7b72f9275\" (UID: \"11cd3f02-9989-4e40-be11-f7e7b72f9275\") "
Nov 02 02:13:42.401507 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.401072    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2584d15c-d3a3-4392-bb37-5b2c650b7cf7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2584d15c-d3a3-4392-bb37-5b2c650b7cf7-plugins-config-volume\") pod \"2584d15c-d3a3-4392-bb37-5b2c650b7cf7\" (UID: \"2584d15c-d3a3-4392-bb37-5b2c650b7cf7\") "
Nov 02 02:13:42.401507 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.401120    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11cd3f02-9989-4e40-be11-f7e7b72f9275-kube-api-access-tx52x\" (UniqueName: \"kubernetes.io/projected/11cd3f02-9989-4e40-be11-f7e7b72f9275-kube-api-access-tx52x\") pod \"11cd3f02-9989-4e40-be11-f7e7b72f9275\" (UID: \"11cd3f02-9989-4e40-be11-f7e7b72f9275\") "
Nov 02 02:13:42.401507 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.401403    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2584d15c-d3a3-4392-bb37-5b2c650b7cf7-kube-api-access-72m28\" (UniqueName: \"kubernetes.io/projected/2584d15c-d3a3-4392-bb37-5b2c650b7cf7-kube-api-access-72m28\") pod \"2584d15c-d3a3-4392-bb37-5b2c650b7cf7\" (UID: \"2584d15c-d3a3-4392-bb37-5b2c650b7cf7\") "
Nov 02 02:13:42.401687 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.401527    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2584d15c-d3a3-4392-bb37-5b2c650b7cf7-plugins-config-volume" pod "2584d15c-d3a3-4392-bb37-5b2c650b7cf7" (UID: "2584d15c-d3a3-4392-bb37-5b2c650b7cf7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:13:42.401687 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.401605    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11cd3f02-9989-4e40-be11-f7e7b72f9275-plugins-config-volume" pod "11cd3f02-9989-4e40-be11-f7e7b72f9275" (UID: "11cd3f02-9989-4e40-be11-f7e7b72f9275"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:13:42.401925 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.401902    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11cd3f02-9989-4e40-be11-f7e7b72f9275-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:42.401925 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.401923    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2584d15c-d3a3-4392-bb37-5b2c650b7cf7-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:42.403797 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.403773    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2584d15c-d3a3-4392-bb37-5b2c650b7cf7-kube-api-access-72m28" pod "2584d15c-d3a3-4392-bb37-5b2c650b7cf7" (UID: "2584d15c-d3a3-4392-bb37-5b2c650b7cf7"). InnerVolumeSpecName "kube-api-access-72m28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:13:42.404129 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.404115    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11cd3f02-9989-4e40-be11-f7e7b72f9275-kube-api-access-tx52x" pod "11cd3f02-9989-4e40-be11-f7e7b72f9275" (UID: "11cd3f02-9989-4e40-be11-f7e7b72f9275"). InnerVolumeSpecName "kube-api-access-tx52x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:13:42.502527 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.502502    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx52x\" (UniqueName: \"kubernetes.io/projected/11cd3f02-9989-4e40-be11-f7e7b72f9275-kube-api-access-tx52x\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:42.502527 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.502525    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72m28\" (UniqueName: \"kubernetes.io/projected/2584d15c-d3a3-4392-bb37-5b2c650b7cf7-kube-api-access-72m28\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:13:42.672528 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.672454    5875 generic.go:356] "Generic (PLEG): container finished" podID="2584d15c-d3a3-4392-bb37-5b2c650b7cf7" containerID="06c7a2f5ca4e3fdd3f6ead4ee54547d54881443ca0c85d542df36f68e77d867d" exitCode=0
Nov 02 02:13:42.672528 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.672507    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp"
Nov 02 02:13:42.672734 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.672537    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp" event={"ID":"2584d15c-d3a3-4392-bb37-5b2c650b7cf7","Type":"ContainerDied","Data":"06c7a2f5ca4e3fdd3f6ead4ee54547d54881443ca0c85d542df36f68e77d867d"}
Nov 02 02:13:42.672734 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.672597    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp" event={"ID":"2584d15c-d3a3-4392-bb37-5b2c650b7cf7","Type":"ContainerDied","Data":"d5007fc033c2df252be089b252c4ff9449f2138c9aaa11b98858a28051d8e22b"}
Nov 02 02:13:42.672734 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.672627    5875 scope.go:122] "RemoveContainer" containerID="06c7a2f5ca4e3fdd3f6ead4ee54547d54881443ca0c85d542df36f68e77d867d"
Nov 02 02:13:42.674025 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.673996    5875 generic.go:356] "Generic (PLEG): container finished" podID="11cd3f02-9989-4e40-be11-f7e7b72f9275" containerID="d9b64da82181e4aa17563fd09b03c46a6a2788c06954e0ae8955eb15e866caee" exitCode=0
Nov 02 02:13:42.674114 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.674054    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8"
Nov 02 02:13:42.674178 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.674055    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8" event={"ID":"11cd3f02-9989-4e40-be11-f7e7b72f9275","Type":"ContainerDied","Data":"d9b64da82181e4aa17563fd09b03c46a6a2788c06954e0ae8955eb15e866caee"}
Nov 02 02:13:42.674267 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.674187    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8" event={"ID":"11cd3f02-9989-4e40-be11-f7e7b72f9275","Type":"ContainerDied","Data":"22ea978c6340f00ec05585315c68c4c9fad552202f5e626019ef3251af223b45"}
Nov 02 02:13:42.682256 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.682234    5875 scope.go:122] "RemoveContainer" containerID="06c7a2f5ca4e3fdd3f6ead4ee54547d54881443ca0c85d542df36f68e77d867d"
Nov 02 02:13:42.682576 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:13:42.682550    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06c7a2f5ca4e3fdd3f6ead4ee54547d54881443ca0c85d542df36f68e77d867d\": container with ID starting with 06c7a2f5ca4e3fdd3f6ead4ee54547d54881443ca0c85d542df36f68e77d867d not found: ID does not exist" containerID="06c7a2f5ca4e3fdd3f6ead4ee54547d54881443ca0c85d542df36f68e77d867d"
Nov 02 02:13:42.682691 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.682580    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06c7a2f5ca4e3fdd3f6ead4ee54547d54881443ca0c85d542df36f68e77d867d"} err="failed to get container status \"06c7a2f5ca4e3fdd3f6ead4ee54547d54881443ca0c85d542df36f68e77d867d\": rpc error: code = NotFound desc = could not find container \"06c7a2f5ca4e3fdd3f6ead4ee54547d54881443ca0c85d542df36f68e77d867d\": container with ID starting with 06c7a2f5ca4e3fdd3f6ead4ee54547d54881443ca0c85d542df36f68e77d867d not found: ID does not exist"
Nov 02 02:13:42.682691 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.682596    5875 scope.go:122] "RemoveContainer" containerID="d9b64da82181e4aa17563fd09b03c46a6a2788c06954e0ae8955eb15e866caee"
Nov 02 02:13:42.692513 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.692495    5875 scope.go:122] "RemoveContainer" containerID="d9b64da82181e4aa17563fd09b03c46a6a2788c06954e0ae8955eb15e866caee"
Nov 02 02:13:42.692781 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:13:42.692764    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9b64da82181e4aa17563fd09b03c46a6a2788c06954e0ae8955eb15e866caee\": container with ID starting with d9b64da82181e4aa17563fd09b03c46a6a2788c06954e0ae8955eb15e866caee not found: ID does not exist" containerID="d9b64da82181e4aa17563fd09b03c46a6a2788c06954e0ae8955eb15e866caee"
Nov 02 02:13:42.692864 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.692785    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b64da82181e4aa17563fd09b03c46a6a2788c06954e0ae8955eb15e866caee"} err="failed to get container status \"d9b64da82181e4aa17563fd09b03c46a6a2788c06954e0ae8955eb15e866caee\": rpc error: code = NotFound desc = could not find container \"d9b64da82181e4aa17563fd09b03c46a6a2788c06954e0ae8955eb15e866caee\": container with ID starting with d9b64da82181e4aa17563fd09b03c46a6a2788c06954e0ae8955eb15e866caee not found: ID does not exist"
Nov 02 02:13:42.839965 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.839925    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp"]
Nov 02 02:13:42.845437 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.845402    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-76837e-7d957fd68c-mc6kp"]
Nov 02 02:13:42.954845 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.954796    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8"]
Nov 02 02:13:42.962958 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:42.962932    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-76837e-7947b4fc6f-pnfq8"]
Nov 02 02:13:43.853344 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:43.853315    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5"]
Nov 02 02:13:43.902065 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:43.902032    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9470a3-mc6w5"]
Nov 02 02:13:44.693661 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:44.693622    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11cd3f02-9989-4e40-be11-f7e7b72f9275" path="/var/lib/kubelet/pods/11cd3f02-9989-4e40-be11-f7e7b72f9275/volumes"
Nov 02 02:13:44.694000 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:44.693987    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2584d15c-d3a3-4392-bb37-5b2c650b7cf7" path="/var/lib/kubelet/pods/2584d15c-d3a3-4392-bb37-5b2c650b7cf7/volumes"
Nov 02 02:13:44.694363 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:44.694349    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5641707e-23f7-4caa-b9cd-485037333d2c" path="/var/lib/kubelet/pods/5641707e-23f7-4caa-b9cd-485037333d2c/volumes"
Nov 02 02:13:50.097187 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:50.097156    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d"]
Nov 02 02:13:50.146448 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:50.146413    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dbe849-lz79d"]
Nov 02 02:13:50.235941 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:50.235906    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:13:50.702492 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:50.702459    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2a2a075-44e5-47a4-bcbf-f04d16246a35" path="/var/lib/kubelet/pods/b2a2a075-44e5-47a4-bcbf-f04d16246a35/volumes"
Nov 02 02:13:56.861369 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:56.861334    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"]
Nov 02 02:13:56.861805 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:13:56.861529    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4" podUID="c1cf6bdd-85db-4669-a694-ac7ab7445792" containerName="istio-proxy" containerID="cri-o://0664e37c3b29a85f87d000542e574c4a421728e5134e5d944353248286603fb2" gracePeriod=30
Nov 02 02:14:02.113825 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.113792    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 02:14:02.157996 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.157930    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-workload-socket\") pod \"c1cf6bdd-85db-4669-a694-ac7ab7445792\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") "
Nov 02 02:14:02.158177 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.158057    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-envoy\") pod \"c1cf6bdd-85db-4669-a694-ac7ab7445792\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") "
Nov 02 02:14:02.158177 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.158100    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-workload-certs\") pod \"c1cf6bdd-85db-4669-a694-ac7ab7445792\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") "
Nov 02 02:14:02.158177 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.158144    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1cf6bdd-85db-4669-a694-ac7ab7445792-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1cf6bdd-85db-4669-a694-ac7ab7445792-istiod-ca-cert\") pod \"c1cf6bdd-85db-4669-a694-ac7ab7445792\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") "
Nov 02 02:14:02.158322 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.158240    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-workload-socket" pod "c1cf6bdd-85db-4669-a694-ac7ab7445792" (UID: "c1cf6bdd-85db-4669-a694-ac7ab7445792"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:02.158322 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.158264    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-data\") pod \"c1cf6bdd-85db-4669-a694-ac7ab7445792\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") "
Nov 02 02:14:02.158322 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.158288    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-credential-socket\") pod \"c1cf6bdd-85db-4669-a694-ac7ab7445792\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") "
Nov 02 02:14:02.158322 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.158312    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-token\" (UniqueName: \"kubernetes.io/projected/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-token\") pod \"c1cf6bdd-85db-4669-a694-ac7ab7445792\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") "
Nov 02 02:14:02.158528 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.158405    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-podinfo\") pod \"c1cf6bdd-85db-4669-a694-ac7ab7445792\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") "
Nov 02 02:14:02.158528 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.158463    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-workload-certs" pod "c1cf6bdd-85db-4669-a694-ac7ab7445792" (UID: "c1cf6bdd-85db-4669-a694-ac7ab7445792"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:02.158528 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.158502    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1cf6bdd-85db-4669-a694-ac7ab7445792-kube-api-access-th8gz\" (UniqueName: \"kubernetes.io/projected/c1cf6bdd-85db-4669-a694-ac7ab7445792-kube-api-access-th8gz\") pod \"c1cf6bdd-85db-4669-a694-ac7ab7445792\" (UID: \"c1cf6bdd-85db-4669-a694-ac7ab7445792\") "
Nov 02 02:14:02.158715 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.158539    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-credential-socket" pod "c1cf6bdd-85db-4669-a694-ac7ab7445792" (UID: "c1cf6bdd-85db-4669-a694-ac7ab7445792"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:02.158715 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.158558    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-data" pod "c1cf6bdd-85db-4669-a694-ac7ab7445792" (UID: "c1cf6bdd-85db-4669-a694-ac7ab7445792"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:02.158715 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.158598    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1cf6bdd-85db-4669-a694-ac7ab7445792-istiod-ca-cert" pod "c1cf6bdd-85db-4669-a694-ac7ab7445792" (UID: "c1cf6bdd-85db-4669-a694-ac7ab7445792"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:14:02.158937 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.158898    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:02.159014 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.158941    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1cf6bdd-85db-4669-a694-ac7ab7445792-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:02.159014 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.158955    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:02.159014 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.158971    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:02.159014 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.158985    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:02.160544 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.160502    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-envoy" pod "c1cf6bdd-85db-4669-a694-ac7ab7445792" (UID: "c1cf6bdd-85db-4669-a694-ac7ab7445792"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:02.160662 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.160612    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-token" pod "c1cf6bdd-85db-4669-a694-ac7ab7445792" (UID: "c1cf6bdd-85db-4669-a694-ac7ab7445792"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:14:02.160714 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.160663    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1cf6bdd-85db-4669-a694-ac7ab7445792-kube-api-access-th8gz" pod "c1cf6bdd-85db-4669-a694-ac7ab7445792" (UID: "c1cf6bdd-85db-4669-a694-ac7ab7445792"). InnerVolumeSpecName "kube-api-access-th8gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:14:02.160805 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.160782    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-podinfo" pod "c1cf6bdd-85db-4669-a694-ac7ab7445792" (UID: "c1cf6bdd-85db-4669-a694-ac7ab7445792"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:14:02.260111 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.260071    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th8gz\" (UniqueName: \"kubernetes.io/projected/c1cf6bdd-85db-4669-a694-ac7ab7445792-kube-api-access-th8gz\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:02.260111 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.260103    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:02.260111 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.260117    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:02.260399 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.260130    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1cf6bdd-85db-4669-a694-ac7ab7445792-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:02.805344 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.805310    5875 generic.go:356] "Generic (PLEG): container finished" podID="c1cf6bdd-85db-4669-a694-ac7ab7445792" containerID="0664e37c3b29a85f87d000542e574c4a421728e5134e5d944353248286603fb2" exitCode=0
Nov 02 02:14:02.805534 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.805354    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"
Nov 02 02:14:02.805534 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.805357    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4" event={"ID":"c1cf6bdd-85db-4669-a694-ac7ab7445792","Type":"ContainerDied","Data":"0664e37c3b29a85f87d000542e574c4a421728e5134e5d944353248286603fb2"}
Nov 02 02:14:02.805534 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.805409    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4" event={"ID":"c1cf6bdd-85db-4669-a694-ac7ab7445792","Type":"ContainerDied","Data":"bad5b920aaf617bf9b2a8871da4c08550ad279d6ac05ff202ffba1361e7a6b88"}
Nov 02 02:14:02.805534 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.805438    5875 scope.go:122] "RemoveContainer" containerID="0664e37c3b29a85f87d000542e574c4a421728e5134e5d944353248286603fb2"
Nov 02 02:14:02.815440 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.815426    5875 scope.go:122] "RemoveContainer" containerID="495f8716113a233c9208a5110f72df2e0f7a3ab94db54ffb77a9e59bb3591d5b"
Nov 02 02:14:02.824035 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.824001    5875 scope.go:122] "RemoveContainer" containerID="0664e37c3b29a85f87d000542e574c4a421728e5134e5d944353248286603fb2"
Nov 02 02:14:02.824320 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:14:02.824302    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0664e37c3b29a85f87d000542e574c4a421728e5134e5d944353248286603fb2\": container with ID starting with 0664e37c3b29a85f87d000542e574c4a421728e5134e5d944353248286603fb2 not found: ID does not exist" containerID="0664e37c3b29a85f87d000542e574c4a421728e5134e5d944353248286603fb2"
Nov 02 02:14:02.824401 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.824337    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0664e37c3b29a85f87d000542e574c4a421728e5134e5d944353248286603fb2"} err="failed to get container status \"0664e37c3b29a85f87d000542e574c4a421728e5134e5d944353248286603fb2\": rpc error: code = NotFound desc = could not find container \"0664e37c3b29a85f87d000542e574c4a421728e5134e5d944353248286603fb2\": container with ID starting with 0664e37c3b29a85f87d000542e574c4a421728e5134e5d944353248286603fb2 not found: ID does not exist"
Nov 02 02:14:02.824401 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.824372    5875 scope.go:122] "RemoveContainer" containerID="495f8716113a233c9208a5110f72df2e0f7a3ab94db54ffb77a9e59bb3591d5b"
Nov 02 02:14:02.824626 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:14:02.824611    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"495f8716113a233c9208a5110f72df2e0f7a3ab94db54ffb77a9e59bb3591d5b\": container with ID starting with 495f8716113a233c9208a5110f72df2e0f7a3ab94db54ffb77a9e59bb3591d5b not found: ID does not exist" containerID="495f8716113a233c9208a5110f72df2e0f7a3ab94db54ffb77a9e59bb3591d5b"
Nov 02 02:14:02.824701 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:02.824630    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495f8716113a233c9208a5110f72df2e0f7a3ab94db54ffb77a9e59bb3591d5b"} err="failed to get container status \"495f8716113a233c9208a5110f72df2e0f7a3ab94db54ffb77a9e59bb3591d5b\": rpc error: code = NotFound desc = could not find container \"495f8716113a233c9208a5110f72df2e0f7a3ab94db54ffb77a9e59bb3591d5b\": container with ID starting with 495f8716113a233c9208a5110f72df2e0f7a3ab94db54ffb77a9e59bb3591d5b not found: ID does not exist"
Nov 02 02:14:03.019257 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:03.019211    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"]
Nov 02 02:14:03.068752 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:03.068725    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dbe849-openshift-default-579655d747-nlfq4"]
Nov 02 02:14:04.669066 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:04.669033    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:14:04.693696 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:04.693669    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1cf6bdd-85db-4669-a694-ac7ab7445792" path="/var/lib/kubelet/pods/c1cf6bdd-85db-4669-a694-ac7ab7445792/volumes"
Nov 02 02:14:11.320553 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:14:11.320514    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:14:12.048943 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:12.048903    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfc6dbfe-wcfp8"]
Nov 02 02:14:12.098310 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:12.098283    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfc6dbfe-wcfp8"]
Nov 02 02:14:12.694360 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:12.694326    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ba1ad30-5e76-4452-821d-ca8309c88af6" path="/var/lib/kubelet/pods/4ba1ad30-5e76-4452-821d-ca8309c88af6/volumes"
Nov 02 02:14:24.909709 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:24.909673    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76837e-6vdrl"]
Nov 02 02:14:24.958769 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:24.958735    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76837e-6vdrl"]
Nov 02 02:14:26.693751 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:26.693711    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8800230-a8a9-4750-9cf1-8921946a48a6" path="/var/lib/kubelet/pods/c8800230-a8a9-4750-9cf1-8921946a48a6/volumes"
Nov 02 02:14:30.316332 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:30.316300    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:14:31.972201 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:31.972171    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"]
Nov 02 02:14:31.972622 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:31.972421    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw" podUID="21514fb5-5be3-4162-bbfe-7d5a46e4a6f9" containerName="istio-proxy" containerID="cri-o://5579e92da538d5d205c67730215069e4c90b2008ade6a46dc7be52f208ba4ad3" gracePeriod=30
Nov 02 02:14:33.011357 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.011320    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr"]
Nov 02 02:14:33.012037 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.012016    5875 cpu_manager.go:414] "containerMap: removing container" podUID="c1cf6bdd-85db-4669-a694-ac7ab7445792" containerName="istio-proxy"
Nov 02 02:14:33.012090 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.012042    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1cf6bdd-85db-4669-a694-ac7ab7445792" containerName="istio-proxy"
Nov 02 02:14:33.012090 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.012083    5875 cpu_manager.go:414] "containerMap: removing container" podUID="d12fc012-f64e-4504-85dc-0bbabf28cb68" containerName="istio-proxy"
Nov 02 02:14:33.012176 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.012091    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d12fc012-f64e-4504-85dc-0bbabf28cb68" containerName="istio-proxy"
Nov 02 02:14:33.012176 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.012109    5875 cpu_manager.go:414] "containerMap: removing container" podUID="c8800230-a8a9-4750-9cf1-8921946a48a6" containerName="revoke-keys"
Nov 02 02:14:33.012176 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.012116    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8800230-a8a9-4750-9cf1-8921946a48a6" containerName="revoke-keys"
Nov 02 02:14:33.012176 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.012137    5875 cpu_manager.go:414] "containerMap: removing container" podUID="2584d15c-d3a3-4392-bb37-5b2c650b7cf7" containerName="payload-pre-processing"
Nov 02 02:14:33.012176 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.012144    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2584d15c-d3a3-4392-bb37-5b2c650b7cf7" containerName="payload-pre-processing"
Nov 02 02:14:33.012176 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.012173    5875 cpu_manager.go:414] "containerMap: removing container" podUID="11cd3f02-9989-4e40-be11-f7e7b72f9275" containerName="payload-processing"
Nov 02 02:14:33.012426 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.012181    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11cd3f02-9989-4e40-be11-f7e7b72f9275" containerName="payload-processing"
Nov 02 02:14:33.012426 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.012199    5875 cpu_manager.go:414] "containerMap: removing container" podUID="d12fc012-f64e-4504-85dc-0bbabf28cb68" containerName="istio-proxy"
Nov 02 02:14:33.012426 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.012206    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d12fc012-f64e-4504-85dc-0bbabf28cb68" containerName="istio-proxy"
Nov 02 02:14:33.012426 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.012346    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="d12fc012-f64e-4504-85dc-0bbabf28cb68" containerName="istio-proxy"
Nov 02 02:14:33.012426 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.012358    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8800230-a8a9-4750-9cf1-8921946a48a6" containerName="revoke-keys"
Nov 02 02:14:33.012426 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.012367    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="2584d15c-d3a3-4392-bb37-5b2c650b7cf7" containerName="payload-pre-processing"
Nov 02 02:14:33.012426 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.012375    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1cf6bdd-85db-4669-a694-ac7ab7445792" containerName="istio-proxy"
Nov 02 02:14:33.012426 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.012381    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="11cd3f02-9989-4e40-be11-f7e7b72f9275" containerName="payload-processing"
Nov 02 02:14:33.012426 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.012391    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1cf6bdd-85db-4669-a694-ac7ab7445792" containerName="istio-proxy"
Nov 02 02:14:33.016358 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.016341    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr"
Nov 02 02:14:33.072974 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.072942    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr"]
Nov 02 02:14:33.115639 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.115600    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80cb04fe-2a4f-4704-b60b-e0b524f4c2ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr\" (UID: \"80cb04fe-2a4f-4704-b60b-e0b524f4c2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr"
Nov 02 02:14:33.115841 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.115778    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb8fk\" (UniqueName: \"kubernetes.io/projected/80cb04fe-2a4f-4704-b60b-e0b524f4c2ab-kube-api-access-lb8fk\") pod \"maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr\" (UID: \"80cb04fe-2a4f-4704-b60b-e0b524f4c2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr"
Nov 02 02:14:33.115926 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.115886    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80cb04fe-2a4f-4704-b60b-e0b524f4c2ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr\" (UID: \"80cb04fe-2a4f-4704-b60b-e0b524f4c2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr"
Nov 02 02:14:33.118950 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.118922    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:14:33.118950 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.118927    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:14:33.217335 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.217299    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80cb04fe-2a4f-4704-b60b-e0b524f4c2ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr\" (UID: \"80cb04fe-2a4f-4704-b60b-e0b524f4c2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr"
Nov 02 02:14:33.217518 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.217351    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb8fk\" (UniqueName: \"kubernetes.io/projected/80cb04fe-2a4f-4704-b60b-e0b524f4c2ab-kube-api-access-lb8fk\") pod \"maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr\" (UID: \"80cb04fe-2a4f-4704-b60b-e0b524f4c2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr"
Nov 02 02:14:33.217518 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.217387    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80cb04fe-2a4f-4704-b60b-e0b524f4c2ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr\" (UID: \"80cb04fe-2a4f-4704-b60b-e0b524f4c2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr"
Nov 02 02:14:33.217959 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.217929    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80cb04fe-2a4f-4704-b60b-e0b524f4c2ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr\" (UID: \"80cb04fe-2a4f-4704-b60b-e0b524f4c2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr"
Nov 02 02:14:33.219882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.219859    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80cb04fe-2a4f-4704-b60b-e0b524f4c2ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr\" (UID: \"80cb04fe-2a4f-4704-b60b-e0b524f4c2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr"
Nov 02 02:14:33.278358 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.278283    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb8fk\" (UniqueName: \"kubernetes.io/projected/80cb04fe-2a4f-4704-b60b-e0b524f4c2ab-kube-api-access-lb8fk\") pod \"maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr\" (UID: \"80cb04fe-2a4f-4704-b60b-e0b524f4c2ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr"
Nov 02 02:14:33.329892 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.329858    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr"
Nov 02 02:14:33.702378 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:33.702354    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr"]
Nov 02 02:14:33.704576 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:14:33.704549    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80cb04fe_2a4f_4704_b60b_e0b524f4c2ab.slice/crio-d9e60ad97daa8be2c2cb596bc94432799573476850cab3e450c5d2dc68b30227 WatchSource:0}: Error finding container d9e60ad97daa8be2c2cb596bc94432799573476850cab3e450c5d2dc68b30227: Status 404 returned error can't find the container with id d9e60ad97daa8be2c2cb596bc94432799573476850cab3e450c5d2dc68b30227
Nov 02 02:14:34.001715 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:34.001676    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr" event={"ID":"80cb04fe-2a4f-4704-b60b-e0b524f4c2ab","Type":"ContainerStarted","Data":"d9e60ad97daa8be2c2cb596bc94432799573476850cab3e450c5d2dc68b30227"}
Nov 02 02:14:35.011968 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:35.011929    5875 generic.go:356] "Generic (PLEG): container finished" podID="80cb04fe-2a4f-4704-b60b-e0b524f4c2ab" containerID="7bac62164b8601941d3d964cb0a86a6876a06178646a5f0296c1bed18baf3127" exitCode=0
Nov 02 02:14:35.012416 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:35.012009    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr" event={"ID":"80cb04fe-2a4f-4704-b60b-e0b524f4c2ab","Type":"ContainerDied","Data":"7bac62164b8601941d3d964cb0a86a6876a06178646a5f0296c1bed18baf3127"}
Nov 02 02:14:36.167056 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:36.167034    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr"
Nov 02 02:14:36.250560 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:36.250527    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80cb04fe-2a4f-4704-b60b-e0b524f4c2ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80cb04fe-2a4f-4704-b60b-e0b524f4c2ab-maas-api-openshift-ca\") pod \"80cb04fe-2a4f-4704-b60b-e0b524f4c2ab\" (UID: \"80cb04fe-2a4f-4704-b60b-e0b524f4c2ab\") "
Nov 02 02:14:36.250782 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:36.250612    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80cb04fe-2a4f-4704-b60b-e0b524f4c2ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80cb04fe-2a4f-4704-b60b-e0b524f4c2ab-maas-api-xks-ca\") pod \"80cb04fe-2a4f-4704-b60b-e0b524f4c2ab\" (UID: \"80cb04fe-2a4f-4704-b60b-e0b524f4c2ab\") "
Nov 02 02:14:36.250782 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:36.250730    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80cb04fe-2a4f-4704-b60b-e0b524f4c2ab-kube-api-access-lb8fk\" (UniqueName: \"kubernetes.io/projected/80cb04fe-2a4f-4704-b60b-e0b524f4c2ab-kube-api-access-lb8fk\") pod \"80cb04fe-2a4f-4704-b60b-e0b524f4c2ab\" (UID: \"80cb04fe-2a4f-4704-b60b-e0b524f4c2ab\") "
Nov 02 02:14:36.250969 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:36.250946    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80cb04fe-2a4f-4704-b60b-e0b524f4c2ab-maas-api-openshift-ca" pod "80cb04fe-2a4f-4704-b60b-e0b524f4c2ab" (UID: "80cb04fe-2a4f-4704-b60b-e0b524f4c2ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:14:36.252963 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:36.252938    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80cb04fe-2a4f-4704-b60b-e0b524f4c2ab-maas-api-xks-ca" pod "80cb04fe-2a4f-4704-b60b-e0b524f4c2ab" (UID: "80cb04fe-2a4f-4704-b60b-e0b524f4c2ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:14:36.253037 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:36.252974    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80cb04fe-2a4f-4704-b60b-e0b524f4c2ab-kube-api-access-lb8fk" pod "80cb04fe-2a4f-4704-b60b-e0b524f4c2ab" (UID: "80cb04fe-2a4f-4704-b60b-e0b524f4c2ab"). InnerVolumeSpecName "kube-api-access-lb8fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:14:36.351919 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:36.351834    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80cb04fe-2a4f-4704-b60b-e0b524f4c2ab-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:36.351919 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:36.351865    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb8fk\" (UniqueName: \"kubernetes.io/projected/80cb04fe-2a4f-4704-b60b-e0b524f4c2ab-kube-api-access-lb8fk\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:36.351919 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:36.351879    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80cb04fe-2a4f-4704-b60b-e0b524f4c2ab-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:37.024791 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.024763    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr"
Nov 02 02:14:37.024932 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.024789    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr" event={"ID":"80cb04fe-2a4f-4704-b60b-e0b524f4c2ab","Type":"ContainerDied","Data":"d9e60ad97daa8be2c2cb596bc94432799573476850cab3e450c5d2dc68b30227"}
Nov 02 02:14:37.024932 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.024844    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9e60ad97daa8be2c2cb596bc94432799573476850cab3e450c5d2dc68b30227"
Nov 02 02:14:37.231014 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.230991    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:14:37.363963 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.363876    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-workload-socket\") pod \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") "
Nov 02 02:14:37.363963 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.363919    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-token\" (UniqueName: \"kubernetes.io/projected/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-token\") pod \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") "
Nov 02 02:14:37.364214 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.363993    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-kube-api-access-rfkvq\" (UniqueName: \"kubernetes.io/projected/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-kube-api-access-rfkvq\") pod \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") "
Nov 02 02:14:37.364214 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.364020    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-workload-certs\") pod \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") "
Nov 02 02:14:37.364214 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.364113    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istiod-ca-cert\") pod \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") "
Nov 02 02:14:37.364214 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.364156    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-podinfo\") pod \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") "
Nov 02 02:14:37.364214 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.364158    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-workload-socket" pod "21514fb5-5be3-4162-bbfe-7d5a46e4a6f9" (UID: "21514fb5-5be3-4162-bbfe-7d5a46e4a6f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:37.364214 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.364187    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-envoy\") pod \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") "
Nov 02 02:14:37.364598 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.364306    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-credential-socket\") pod \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") "
Nov 02 02:14:37.364598 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.364336    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-data\") pod \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\" (UID: \"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9\") "
Nov 02 02:14:37.364598 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.364377    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-workload-certs" pod "21514fb5-5be3-4162-bbfe-7d5a46e4a6f9" (UID: "21514fb5-5be3-4162-bbfe-7d5a46e4a6f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:37.364598 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.364491    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istiod-ca-cert" pod "21514fb5-5be3-4162-bbfe-7d5a46e4a6f9" (UID: "21514fb5-5be3-4162-bbfe-7d5a46e4a6f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:14:37.364824 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.364663    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-credential-socket" pod "21514fb5-5be3-4162-bbfe-7d5a46e4a6f9" (UID: "21514fb5-5be3-4162-bbfe-7d5a46e4a6f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:37.364886 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.364840    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:37.364886 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.364861    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:37.364886 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.364876    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:37.365055 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.364890    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:37.365116 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.365051    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-data" pod "21514fb5-5be3-4162-bbfe-7d5a46e4a6f9" (UID: "21514fb5-5be3-4162-bbfe-7d5a46e4a6f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:37.366499 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.366476    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-kube-api-access-rfkvq" pod "21514fb5-5be3-4162-bbfe-7d5a46e4a6f9" (UID: "21514fb5-5be3-4162-bbfe-7d5a46e4a6f9"). InnerVolumeSpecName "kube-api-access-rfkvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:14:37.367110 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.367086    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-podinfo" pod "21514fb5-5be3-4162-bbfe-7d5a46e4a6f9" (UID: "21514fb5-5be3-4162-bbfe-7d5a46e4a6f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:14:37.367110 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.367084    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-envoy" pod "21514fb5-5be3-4162-bbfe-7d5a46e4a6f9" (UID: "21514fb5-5be3-4162-bbfe-7d5a46e4a6f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:37.367317 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.367283    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-token" pod "21514fb5-5be3-4162-bbfe-7d5a46e4a6f9" (UID: "21514fb5-5be3-4162-bbfe-7d5a46e4a6f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:14:37.404831 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.404798    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"]
Nov 02 02:14:37.466468 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.466434    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:37.466468 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.466463    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:37.466468 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.466473    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:37.466703 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.466481    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:37.466703 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:37.466490    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfkvq\" (UniqueName: \"kubernetes.io/projected/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9-kube-api-access-rfkvq\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:38.032903 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:38.032868    5875 generic.go:356] "Generic (PLEG): container finished" podID="21514fb5-5be3-4162-bbfe-7d5a46e4a6f9" containerID="5579e92da538d5d205c67730215069e4c90b2008ade6a46dc7be52f208ba4ad3" exitCode=0
Nov 02 02:14:38.033081 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:38.032920    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw" event={"ID":"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9","Type":"ContainerDied","Data":"5579e92da538d5d205c67730215069e4c90b2008ade6a46dc7be52f208ba4ad3"}
Nov 02 02:14:38.033081 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:38.032929    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"
Nov 02 02:14:38.033081 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:38.032971    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw" event={"ID":"21514fb5-5be3-4162-bbfe-7d5a46e4a6f9","Type":"ContainerDied","Data":"d5e3d18b9ebf424c7be3c3fbe88c57a6681a7871a83dce311bb9a982398cb924"}
Nov 02 02:14:38.033081 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:38.033001    5875 scope.go:122] "RemoveContainer" containerID="5579e92da538d5d205c67730215069e4c90b2008ade6a46dc7be52f208ba4ad3"
Nov 02 02:14:38.044379 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:38.044356    5875 scope.go:122] "RemoveContainer" containerID="91ad76a6027fa6435bdfdee6f1de3b16868f36c038e5274351c150a1829f8bbc"
Nov 02 02:14:38.055620 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:38.055591    5875 scope.go:122] "RemoveContainer" containerID="5579e92da538d5d205c67730215069e4c90b2008ade6a46dc7be52f208ba4ad3"
Nov 02 02:14:38.055969 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:14:38.055930    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5579e92da538d5d205c67730215069e4c90b2008ade6a46dc7be52f208ba4ad3\": container with ID starting with 5579e92da538d5d205c67730215069e4c90b2008ade6a46dc7be52f208ba4ad3 not found: ID does not exist" containerID="5579e92da538d5d205c67730215069e4c90b2008ade6a46dc7be52f208ba4ad3"
Nov 02 02:14:38.056036 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:38.055981    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5579e92da538d5d205c67730215069e4c90b2008ade6a46dc7be52f208ba4ad3"} err="failed to get container status \"5579e92da538d5d205c67730215069e4c90b2008ade6a46dc7be52f208ba4ad3\": rpc error: code = NotFound desc = could not find container \"5579e92da538d5d205c67730215069e4c90b2008ade6a46dc7be52f208ba4ad3\": container with ID starting with 5579e92da538d5d205c67730215069e4c90b2008ade6a46dc7be52f208ba4ad3 not found: ID does not exist"
Nov 02 02:14:38.056036 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:38.056001    5875 scope.go:122] "RemoveContainer" containerID="91ad76a6027fa6435bdfdee6f1de3b16868f36c038e5274351c150a1829f8bbc"
Nov 02 02:14:38.056323 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:14:38.056300    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91ad76a6027fa6435bdfdee6f1de3b16868f36c038e5274351c150a1829f8bbc\": container with ID starting with 91ad76a6027fa6435bdfdee6f1de3b16868f36c038e5274351c150a1829f8bbc not found: ID does not exist" containerID="91ad76a6027fa6435bdfdee6f1de3b16868f36c038e5274351c150a1829f8bbc"
Nov 02 02:14:38.056377 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:38.056328    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91ad76a6027fa6435bdfdee6f1de3b16868f36c038e5274351c150a1829f8bbc"} err="failed to get container status \"91ad76a6027fa6435bdfdee6f1de3b16868f36c038e5274351c150a1829f8bbc\": rpc error: code = NotFound desc = could not find container \"91ad76a6027fa6435bdfdee6f1de3b16868f36c038e5274351c150a1829f8bbc\": container with ID starting with 91ad76a6027fa6435bdfdee6f1de3b16868f36c038e5274351c150a1829f8bbc not found: ID does not exist"
Nov 02 02:14:38.197334 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:38.197298    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"]
Nov 02 02:14:38.204261 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:38.204208    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-76837e-openshift-default-795bd7788c-czsgw"]
Nov 02 02:14:38.695445 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:38.695411    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21514fb5-5be3-4162-bbfe-7d5a46e4a6f9" path="/var/lib/kubelet/pods/21514fb5-5be3-4162-bbfe-7d5a46e4a6f9/volumes"
Nov 02 02:14:41.318390 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:14:41.318301    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:14:44.490542 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:44.490509    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822"]
Nov 02 02:14:44.596354 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:44.596319    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr"]
Nov 02 02:14:49.491716 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:49.491660    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822" podUID="90c6117a-dbee-4707-ba61-2e6971c1d52f" containerName="payload-processing" containerID="cri-o://7afe8cd641902e780ff2e240c8e0d2e12b12d59bb147494b10618c3bda4f9ecb" gracePeriod=25
Nov 02 02:14:49.597017 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:49.596976    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr" podUID="70879f63-ab1e-41ba-83e8-4df35dc40d1d" containerName="payload-pre-processing" containerID="cri-o://97c2c7497e33e3cf304f688087cc7e18c1005318b4710764e956e215a399a8ac" gracePeriod=25
Nov 02 02:14:52.430721 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:52.430672    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main" containerID="cri-o://b0f63abbe87826d341bb9f37171af49bf66235fa974b6b5abbcaa73ac649fe00" gracePeriod=45
Nov 02 02:14:52.695988 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:52.695923    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 02:14:52.732984 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:52.732948    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-tmp-dir\") pod \"81073143-dea0-4ec5-844f-f58d3b0f8d33\" (UID: \"81073143-dea0-4ec5-844f-f58d3b0f8d33\") "
Nov 02 02:14:52.733159 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:52.733006    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-model-cache\") pod \"81073143-dea0-4ec5-844f-f58d3b0f8d33\" (UID: \"81073143-dea0-4ec5-844f-f58d3b0f8d33\") "
Nov 02 02:14:52.733159 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:52.733093    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81073143-dea0-4ec5-844f-f58d3b0f8d33-kube-api-access-r2bfm\" (UniqueName: \"kubernetes.io/projected/81073143-dea0-4ec5-844f-f58d3b0f8d33-kube-api-access-r2bfm\") pod \"81073143-dea0-4ec5-844f-f58d3b0f8d33\" (UID: \"81073143-dea0-4ec5-844f-f58d3b0f8d33\") "
Nov 02 02:14:52.733159 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:52.733141    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-home\" (UniqueName: \"kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-home\") pod \"81073143-dea0-4ec5-844f-f58d3b0f8d33\" (UID: \"81073143-dea0-4ec5-844f-f58d3b0f8d33\") "
Nov 02 02:14:52.733373 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:52.733193    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-tmp-dir" pod "81073143-dea0-4ec5-844f-f58d3b0f8d33" (UID: "81073143-dea0-4ec5-844f-f58d3b0f8d33"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:52.733373 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:52.733200    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81073143-dea0-4ec5-844f-f58d3b0f8d33-tls-certs\" (UniqueName: \"kubernetes.io/secret/81073143-dea0-4ec5-844f-f58d3b0f8d33-tls-certs\") pod \"81073143-dea0-4ec5-844f-f58d3b0f8d33\" (UID: \"81073143-dea0-4ec5-844f-f58d3b0f8d33\") "
Nov 02 02:14:52.733373 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:52.733307    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-dshm\" (UniqueName: \"kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-dshm\") pod \"81073143-dea0-4ec5-844f-f58d3b0f8d33\" (UID: \"81073143-dea0-4ec5-844f-f58d3b0f8d33\") "
Nov 02 02:14:52.733373 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:52.733352    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-model-cache" pod "81073143-dea0-4ec5-844f-f58d3b0f8d33" (UID: "81073143-dea0-4ec5-844f-f58d3b0f8d33"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:52.733593 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:52.733445    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-home" pod "81073143-dea0-4ec5-844f-f58d3b0f8d33" (UID: "81073143-dea0-4ec5-844f-f58d3b0f8d33"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:52.733785 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:52.733760    5875 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-tmp-dir\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:52.733931 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:52.733842    5875 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-model-cache\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:52.733931 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:52.733859    5875 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-home\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:52.735543 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:52.735515    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81073143-dea0-4ec5-844f-f58d3b0f8d33-tls-certs" pod "81073143-dea0-4ec5-844f-f58d3b0f8d33" (UID: "81073143-dea0-4ec5-844f-f58d3b0f8d33"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:14:52.735543 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:52.735529    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81073143-dea0-4ec5-844f-f58d3b0f8d33-kube-api-access-r2bfm" pod "81073143-dea0-4ec5-844f-f58d3b0f8d33" (UID: "81073143-dea0-4ec5-844f-f58d3b0f8d33"). InnerVolumeSpecName "kube-api-access-r2bfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:14:52.735680 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:52.735544    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-dshm" pod "81073143-dea0-4ec5-844f-f58d3b0f8d33" (UID: "81073143-dea0-4ec5-844f-f58d3b0f8d33"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:52.835418 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:52.835379    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2bfm\" (UniqueName: \"kubernetes.io/projected/81073143-dea0-4ec5-844f-f58d3b0f8d33-kube-api-access-r2bfm\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:52.835418 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:52.835413    5875 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81073143-dea0-4ec5-844f-f58d3b0f8d33-tls-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:52.835418 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:52.835426    5875 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81073143-dea0-4ec5-844f-f58d3b0f8d33-dshm\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:52.984517 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:52.984478    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr" podUID="70879f63-ab1e-41ba-83e8-4df35dc40d1d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:14:52.984709 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:52.984488    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822" podUID="90c6117a-dbee-4707-ba61-2e6971c1d52f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:14:53.125571 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:53.125542    5875 generic.go:356] "Generic (PLEG): container finished" podID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerID="b0f63abbe87826d341bb9f37171af49bf66235fa974b6b5abbcaa73ac649fe00" exitCode=0
Nov 02 02:14:53.125756 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:53.125591    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"
Nov 02 02:14:53.125756 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:53.125633    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" event={"ID":"81073143-dea0-4ec5-844f-f58d3b0f8d33","Type":"ContainerDied","Data":"b0f63abbe87826d341bb9f37171af49bf66235fa974b6b5abbcaa73ac649fe00"}
Nov 02 02:14:53.125756 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:53.125674    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8" event={"ID":"81073143-dea0-4ec5-844f-f58d3b0f8d33","Type":"ContainerDied","Data":"5a3ffae82b93f197704068530896bf8064eb877c991d5007a0628a4b9909abcc"}
Nov 02 02:14:53.125756 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:53.125691    5875 scope.go:122] "RemoveContainer" containerID="b0f63abbe87826d341bb9f37171af49bf66235fa974b6b5abbcaa73ac649fe00"
Nov 02 02:14:53.136748 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:53.136734    5875 scope.go:122] "RemoveContainer" containerID="b0f63abbe87826d341bb9f37171af49bf66235fa974b6b5abbcaa73ac649fe00"
Nov 02 02:14:53.136952 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:14:53.136936    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f63abbe87826d341bb9f37171af49bf66235fa974b6b5abbcaa73ac649fe00\": container with ID starting with b0f63abbe87826d341bb9f37171af49bf66235fa974b6b5abbcaa73ac649fe00 not found: ID does not exist" containerID="b0f63abbe87826d341bb9f37171af49bf66235fa974b6b5abbcaa73ac649fe00"
Nov 02 02:14:53.137019 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:53.136959    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f63abbe87826d341bb9f37171af49bf66235fa974b6b5abbcaa73ac649fe00"} err="failed to get container status \"b0f63abbe87826d341bb9f37171af49bf66235fa974b6b5abbcaa73ac649fe00\": rpc error: code = NotFound desc = could not find container \"b0f63abbe87826d341bb9f37171af49bf66235fa974b6b5abbcaa73ac649fe00\": container with ID starting with b0f63abbe87826d341bb9f37171af49bf66235fa974b6b5abbcaa73ac649fe00 not found: ID does not exist"
Nov 02 02:14:53.289480 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:53.289401    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"]
Nov 02 02:14:53.293671 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:53.293644    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-c62671/distinct-2-w2-c62671-kserve-5c76586965-dd5r8"]
Nov 02 02:14:54.693845 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:54.693814    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" path="/var/lib/kubelet/pods/81073143-dea0-4ec5-844f-f58d3b0f8d33/volumes"
Nov 02 02:14:58.259099 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.259067    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c96f480-h5lwz"]
Nov 02 02:14:58.259680 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.259639    5875 cpu_manager.go:414] "containerMap: removing container" podUID="21514fb5-5be3-4162-bbfe-7d5a46e4a6f9" containerName="istio-proxy"
Nov 02 02:14:58.259680 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.259653    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21514fb5-5be3-4162-bbfe-7d5a46e4a6f9" containerName="istio-proxy"
Nov 02 02:14:58.259680 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.259676    5875 cpu_manager.go:414] "containerMap: removing container" podUID="c1cf6bdd-85db-4669-a694-ac7ab7445792" containerName="istio-proxy"
Nov 02 02:14:58.259680 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.259681    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1cf6bdd-85db-4669-a694-ac7ab7445792" containerName="istio-proxy"
Nov 02 02:14:58.259890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.259692    5875 cpu_manager.go:414] "containerMap: removing container" podUID="21514fb5-5be3-4162-bbfe-7d5a46e4a6f9" containerName="istio-proxy"
Nov 02 02:14:58.259890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.259697    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21514fb5-5be3-4162-bbfe-7d5a46e4a6f9" containerName="istio-proxy"
Nov 02 02:14:58.259890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.259704    5875 cpu_manager.go:414] "containerMap: removing container" podUID="80cb04fe-2a4f-4704-b60b-e0b524f4c2ab" containerName="revoke-keys"
Nov 02 02:14:58.259890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.259709    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80cb04fe-2a4f-4704-b60b-e0b524f4c2ab" containerName="revoke-keys"
Nov 02 02:14:58.259890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.259724    5875 cpu_manager.go:414] "containerMap: removing container" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main"
Nov 02 02:14:58.259890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.259728    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main"
Nov 02 02:14:58.259890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.259819    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="21514fb5-5be3-4162-bbfe-7d5a46e4a6f9" containerName="istio-proxy"
Nov 02 02:14:58.259890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.259827    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="80cb04fe-2a4f-4704-b60b-e0b524f4c2ab" containerName="revoke-keys"
Nov 02 02:14:58.259890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.259835    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="81073143-dea0-4ec5-844f-f58d3b0f8d33" containerName="main"
Nov 02 02:14:58.259890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.259841    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="d12fc012-f64e-4504-85dc-0bbabf28cb68" containerName="istio-proxy"
Nov 02 02:14:58.263257 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.263236    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c96f480-h5lwz"
Nov 02 02:14:58.296108 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.296080    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49894b1d-dea5-4978-b51f-b3e6f80b1326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c96f480-h5lwz\" (UID: \"49894b1d-dea5-4978-b51f-b3e6f80b1326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c96f480-h5lwz"
Nov 02 02:14:58.296242 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.296128    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l6wl\" (UniqueName: \"kubernetes.io/projected/49894b1d-dea5-4978-b51f-b3e6f80b1326-kube-api-access-8l6wl\") pod \"maas-api-revoke-sub-1c96f480-h5lwz\" (UID: \"49894b1d-dea5-4978-b51f-b3e6f80b1326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c96f480-h5lwz"
Nov 02 02:14:58.296304 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.296260    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49894b1d-dea5-4978-b51f-b3e6f80b1326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c96f480-h5lwz\" (UID: \"49894b1d-dea5-4978-b51f-b3e6f80b1326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c96f480-h5lwz"
Nov 02 02:14:58.366084 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.366051    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:14:58.366261 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.366162    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:14:58.369717 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.369695    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c96f480-h5lwz"]
Nov 02 02:14:58.397451 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.397423    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49894b1d-dea5-4978-b51f-b3e6f80b1326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c96f480-h5lwz\" (UID: \"49894b1d-dea5-4978-b51f-b3e6f80b1326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c96f480-h5lwz"
Nov 02 02:14:58.397592 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.397470    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l6wl\" (UniqueName: \"kubernetes.io/projected/49894b1d-dea5-4978-b51f-b3e6f80b1326-kube-api-access-8l6wl\") pod \"maas-api-revoke-sub-1c96f480-h5lwz\" (UID: \"49894b1d-dea5-4978-b51f-b3e6f80b1326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c96f480-h5lwz"
Nov 02 02:14:58.397592 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.397520    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49894b1d-dea5-4978-b51f-b3e6f80b1326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c96f480-h5lwz\" (UID: \"49894b1d-dea5-4978-b51f-b3e6f80b1326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c96f480-h5lwz"
Nov 02 02:14:58.398300 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.398210    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49894b1d-dea5-4978-b51f-b3e6f80b1326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c96f480-h5lwz\" (UID: \"49894b1d-dea5-4978-b51f-b3e6f80b1326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c96f480-h5lwz"
Nov 02 02:14:58.400029 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.400002    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49894b1d-dea5-4978-b51f-b3e6f80b1326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c96f480-h5lwz\" (UID: \"49894b1d-dea5-4978-b51f-b3e6f80b1326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c96f480-h5lwz"
Nov 02 02:14:58.458792 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.458760    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l6wl\" (UniqueName: \"kubernetes.io/projected/49894b1d-dea5-4978-b51f-b3e6f80b1326-kube-api-access-8l6wl\") pod \"maas-api-revoke-sub-1c96f480-h5lwz\" (UID: \"49894b1d-dea5-4978-b51f-b3e6f80b1326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c96f480-h5lwz"
Nov 02 02:14:58.576078 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.575991    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c96f480-h5lwz"
Nov 02 02:14:58.949982 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:58.949958    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c96f480-h5lwz"]
Nov 02 02:14:58.951998 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:14:58.951967    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49894b1d_dea5_4978_b51f_b3e6f80b1326.slice/crio-ea83119b0d7da99f62876905624e41a77dd90a49cf4dde0170743b7c2974a2cf WatchSource:0}: Error finding container ea83119b0d7da99f62876905624e41a77dd90a49cf4dde0170743b7c2974a2cf: Status 404 returned error can't find the container with id ea83119b0d7da99f62876905624e41a77dd90a49cf4dde0170743b7c2974a2cf
Nov 02 02:14:59.188127 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:14:59.188076    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c96f480-h5lwz" event={"ID":"49894b1d-dea5-4978-b51f-b3e6f80b1326","Type":"ContainerStarted","Data":"ea83119b0d7da99f62876905624e41a77dd90a49cf4dde0170743b7c2974a2cf"}
Nov 02 02:15:00.157274 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:00.157198    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs"]
Nov 02 02:15:00.157833 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:00.157807    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="21514fb5-5be3-4162-bbfe-7d5a46e4a6f9" containerName="istio-proxy"
Nov 02 02:15:00.161172 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:00.161152    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs"
Nov 02 02:15:00.163391 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:00.163368    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb"]
Nov 02 02:15:00.166959 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:00.166941    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb"
Nov 02 02:15:00.195183 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:00.195155    5875 generic.go:356] "Generic (PLEG): container finished" podID="49894b1d-dea5-4978-b51f-b3e6f80b1326" containerID="b5d0849624a9ff1cf546353c6a690126d4f54e4c837cb938387a72be4c83ef27" exitCode=0
Nov 02 02:15:00.195307 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:00.195262    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c96f480-h5lwz" event={"ID":"49894b1d-dea5-4978-b51f-b3e6f80b1326","Type":"ContainerDied","Data":"b5d0849624a9ff1cf546353c6a690126d4f54e4c837cb938387a72be4c83ef27"}
Nov 02 02:15:00.216663 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:00.216634    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jlvd\" (UniqueName: \"kubernetes.io/projected/44c159c0-914d-49b3-a501-bb0cdfc31e9d-kube-api-access-5jlvd\") pod \"maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs\" (UID: \"44c159c0-914d-49b3-a501-bb0cdfc31e9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs"
Nov 02 02:15:00.216766 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:00.216702    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31bbc102-ee95-4c55-b953-c41f00fd4831-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb\" (UID: \"31bbc102-ee95-4c55-b953-c41f00fd4831\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb"
Nov 02 02:15:00.216818 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:00.216800    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs2pj\" (UniqueName: \"kubernetes.io/projected/31bbc102-ee95-4c55-b953-c41f00fd4831-kube-api-access-rs2pj\") pod \"maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb\" (UID: \"31bbc102-ee95-4c55-b953-c41f00fd4831\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb"
Nov 02 02:15:00.216857 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:00.216845    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44c159c0-914d-49b3-a501-bb0cdfc31e9d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs\" (UID: \"44c159c0-914d-49b3-a501-bb0cdfc31e9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs"
Nov 02 02:15:00.220997 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:00.220969    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs"]
Nov 02 02:15:00.270485 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:00.270448    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb"]
Nov 02 02:15:00.317738 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:00.317702    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44c159c0-914d-49b3-a501-bb0cdfc31e9d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs\" (UID: \"44c159c0-914d-49b3-a501-bb0cdfc31e9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs"
Nov 02 02:15:00.317897 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:00.317851    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jlvd\" (UniqueName: \"kubernetes.io/projected/44c159c0-914d-49b3-a501-bb0cdfc31e9d-kube-api-access-5jlvd\") pod \"maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs\" (UID: \"44c159c0-914d-49b3-a501-bb0cdfc31e9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs"
Nov 02 02:15:00.317971 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:00.317955    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31bbc102-ee95-4c55-b953-c41f00fd4831-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb\" (UID: \"31bbc102-ee95-4c55-b953-c41f00fd4831\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb"
Nov 02 02:15:00.318029 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:00.318013    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs2pj\" (UniqueName: \"kubernetes.io/projected/31bbc102-ee95-4c55-b953-c41f00fd4831-kube-api-access-rs2pj\") pod \"maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb\" (UID: \"31bbc102-ee95-4c55-b953-c41f00fd4831\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb"
Nov 02 02:15:00.318403 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:00.318387    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44c159c0-914d-49b3-a501-bb0cdfc31e9d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs\" (UID: \"44c159c0-914d-49b3-a501-bb0cdfc31e9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs"
Nov 02 02:15:00.318611 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:00.318593    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31bbc102-ee95-4c55-b953-c41f00fd4831-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb\" (UID: \"31bbc102-ee95-4c55-b953-c41f00fd4831\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb"
Nov 02 02:15:00.378364 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:00.378336    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jlvd\" (UniqueName: \"kubernetes.io/projected/44c159c0-914d-49b3-a501-bb0cdfc31e9d-kube-api-access-5jlvd\") pod \"maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs\" (UID: \"44c159c0-914d-49b3-a501-bb0cdfc31e9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs"
Nov 02 02:15:00.378364 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:00.378363    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs2pj\" (UniqueName: \"kubernetes.io/projected/31bbc102-ee95-4c55-b953-c41f00fd4831-kube-api-access-rs2pj\") pod \"maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb\" (UID: \"31bbc102-ee95-4c55-b953-c41f00fd4831\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb"
Nov 02 02:15:00.474900 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:00.474864    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs"
Nov 02 02:15:00.482064 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:00.482045    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb"
Nov 02 02:15:00.853429 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:00.853404    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs"]
Nov 02 02:15:00.855310 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:15:00.855280    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c159c0_914d_49b3_a501_bb0cdfc31e9d.slice/crio-6c9ad30c2399e89dba86ece1b0b313b50c146727dc7202763fda0af814e6c585 WatchSource:0}: Error finding container 6c9ad30c2399e89dba86ece1b0b313b50c146727dc7202763fda0af814e6c585: Status 404 returned error can't find the container with id 6c9ad30c2399e89dba86ece1b0b313b50c146727dc7202763fda0af814e6c585
Nov 02 02:15:00.903381 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:00.903358    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb"]
Nov 02 02:15:00.905257 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:15:00.905211    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31bbc102_ee95_4c55_b953_c41f00fd4831.slice/crio-8a47e715e107d5357f1f7d20cd7d832cae680539cc0ec3959f862831acacecae WatchSource:0}: Error finding container 8a47e715e107d5357f1f7d20cd7d832cae680539cc0ec3959f862831acacecae: Status 404 returned error can't find the container with id 8a47e715e107d5357f1f7d20cd7d832cae680539cc0ec3959f862831acacecae
Nov 02 02:15:01.203611 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:01.203570    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs" event={"ID":"44c159c0-914d-49b3-a501-bb0cdfc31e9d","Type":"ContainerStarted","Data":"6f0831e8edd1953433181bdff763056718b25f31f31e3fac5b7def509a26c9d9"}
Nov 02 02:15:01.204037 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:01.203690    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs" event={"ID":"44c159c0-914d-49b3-a501-bb0cdfc31e9d","Type":"ContainerStarted","Data":"6c9ad30c2399e89dba86ece1b0b313b50c146727dc7202763fda0af814e6c585"}
Nov 02 02:15:01.204953 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:01.204930    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb" event={"ID":"31bbc102-ee95-4c55-b953-c41f00fd4831","Type":"ContainerStarted","Data":"8a47e715e107d5357f1f7d20cd7d832cae680539cc0ec3959f862831acacecae"}
Nov 02 02:15:01.318950 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:01.318901    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs" podStartSLOduration=1.318876789 podStartE2EDuration="1.318876789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:15:01.317439301 +0000 UTC m=+2548.758475834" watchObservedRunningTime="2026-11-02 02:15:01.318876789 +0000 UTC m=+2548.759913286"
Nov 02 02:15:01.333803 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:01.333778    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c96f480-h5lwz"
Nov 02 02:15:01.431834 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:01.431756    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49894b1d-dea5-4978-b51f-b3e6f80b1326-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49894b1d-dea5-4978-b51f-b3e6f80b1326-maas-api-openshift-ca\") pod \"49894b1d-dea5-4978-b51f-b3e6f80b1326\" (UID: \"49894b1d-dea5-4978-b51f-b3e6f80b1326\") "
Nov 02 02:15:01.431989 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:01.431873    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49894b1d-dea5-4978-b51f-b3e6f80b1326-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49894b1d-dea5-4978-b51f-b3e6f80b1326-maas-api-xks-ca\") pod \"49894b1d-dea5-4978-b51f-b3e6f80b1326\" (UID: \"49894b1d-dea5-4978-b51f-b3e6f80b1326\") "
Nov 02 02:15:01.431989 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:01.431898    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49894b1d-dea5-4978-b51f-b3e6f80b1326-kube-api-access-8l6wl\" (UniqueName: \"kubernetes.io/projected/49894b1d-dea5-4978-b51f-b3e6f80b1326-kube-api-access-8l6wl\") pod \"49894b1d-dea5-4978-b51f-b3e6f80b1326\" (UID: \"49894b1d-dea5-4978-b51f-b3e6f80b1326\") "
Nov 02 02:15:01.432170 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:01.432140    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49894b1d-dea5-4978-b51f-b3e6f80b1326-maas-api-openshift-ca" pod "49894b1d-dea5-4978-b51f-b3e6f80b1326" (UID: "49894b1d-dea5-4978-b51f-b3e6f80b1326"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:15:01.434021 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:01.433995    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49894b1d-dea5-4978-b51f-b3e6f80b1326-maas-api-xks-ca" pod "49894b1d-dea5-4978-b51f-b3e6f80b1326" (UID: "49894b1d-dea5-4978-b51f-b3e6f80b1326"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:15:01.434125 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:01.434067    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49894b1d-dea5-4978-b51f-b3e6f80b1326-kube-api-access-8l6wl" pod "49894b1d-dea5-4978-b51f-b3e6f80b1326" (UID: "49894b1d-dea5-4978-b51f-b3e6f80b1326"). InnerVolumeSpecName "kube-api-access-8l6wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:15:01.533235 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:01.533188    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49894b1d-dea5-4978-b51f-b3e6f80b1326-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:01.533235 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:01.533233    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l6wl\" (UniqueName: \"kubernetes.io/projected/49894b1d-dea5-4978-b51f-b3e6f80b1326-kube-api-access-8l6wl\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:01.533440 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:01.533245    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49894b1d-dea5-4978-b51f-b3e6f80b1326-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:02.213135 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:02.213099    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c96f480-h5lwz"
Nov 02 02:15:02.213616 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:02.213126    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c96f480-h5lwz" event={"ID":"49894b1d-dea5-4978-b51f-b3e6f80b1326","Type":"ContainerDied","Data":"ea83119b0d7da99f62876905624e41a77dd90a49cf4dde0170743b7c2974a2cf"}
Nov 02 02:15:02.213616 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:02.213188    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea83119b0d7da99f62876905624e41a77dd90a49cf4dde0170743b7c2974a2cf"
Nov 02 02:15:02.214897 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:02.214863    5875 generic.go:356] "Generic (PLEG): container finished" podID="44c159c0-914d-49b3-a501-bb0cdfc31e9d" containerID="6f0831e8edd1953433181bdff763056718b25f31f31e3fac5b7def509a26c9d9" exitCode=60
Nov 02 02:15:02.215017 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:02.214931    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs" event={"ID":"44c159c0-914d-49b3-a501-bb0cdfc31e9d","Type":"ContainerDied","Data":"6f0831e8edd1953433181bdff763056718b25f31f31e3fac5b7def509a26c9d9"}
Nov 02 02:15:02.215275 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:02.215259    5875 scope.go:122] "RemoveContainer" containerID="6f0831e8edd1953433181bdff763056718b25f31f31e3fac5b7def509a26c9d9"
Nov 02 02:15:02.216768 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:02.216721    5875 generic.go:356] "Generic (PLEG): container finished" podID="31bbc102-ee95-4c55-b953-c41f00fd4831" containerID="2277119685c2a29e4c02eadabca6d4b6da1718816103f8f86f4b0fab8be0aeba" exitCode=60
Nov 02 02:15:02.216768 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:02.216743    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb" event={"ID":"31bbc102-ee95-4c55-b953-c41f00fd4831","Type":"ContainerDied","Data":"2277119685c2a29e4c02eadabca6d4b6da1718816103f8f86f4b0fab8be0aeba"}
Nov 02 02:15:02.217017 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:02.217002    5875 scope.go:122] "RemoveContainer" containerID="2277119685c2a29e4c02eadabca6d4b6da1718816103f8f86f4b0fab8be0aeba"
Nov 02 02:15:02.984162 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:02.984128    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr" podUID="70879f63-ab1e-41ba-83e8-4df35dc40d1d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:15:02.984366 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:02.984134    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822" podUID="90c6117a-dbee-4707-ba61-2e6971c1d52f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:15:03.224557 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:03.224516    5875 generic.go:356] "Generic (PLEG): container finished" podID="44c159c0-914d-49b3-a501-bb0cdfc31e9d" containerID="764e843bb30f4189a81a61fb70d867b5b6844a56f53d11dcd1477e2742bab0af" exitCode=60
Nov 02 02:15:03.225002 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:03.224597    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs" event={"ID":"44c159c0-914d-49b3-a501-bb0cdfc31e9d","Type":"ContainerDied","Data":"764e843bb30f4189a81a61fb70d867b5b6844a56f53d11dcd1477e2742bab0af"}
Nov 02 02:15:03.225002 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:03.224656    5875 scope.go:122] "RemoveContainer" containerID="6f0831e8edd1953433181bdff763056718b25f31f31e3fac5b7def509a26c9d9"
Nov 02 02:15:03.225002 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:03.224934    5875 scope.go:122] "RemoveContainer" containerID="764e843bb30f4189a81a61fb70d867b5b6844a56f53d11dcd1477e2742bab0af"
Nov 02 02:15:03.225309 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:15:03.225287    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs_odh-ai-gateway-infra(44c159c0-914d-49b3-a501-bb0cdfc31e9d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs" podUID="44c159c0-914d-49b3-a501-bb0cdfc31e9d"
Nov 02 02:15:03.227123 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:03.227099    5875 generic.go:356] "Generic (PLEG): container finished" podID="31bbc102-ee95-4c55-b953-c41f00fd4831" containerID="9c7e98a17a103e727d7c93c08e8722f96709283a68b409105a3e259fffc7d3d1" exitCode=60
Nov 02 02:15:03.227285 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:03.227168    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb" event={"ID":"31bbc102-ee95-4c55-b953-c41f00fd4831","Type":"ContainerDied","Data":"9c7e98a17a103e727d7c93c08e8722f96709283a68b409105a3e259fffc7d3d1"}
Nov 02 02:15:03.227474 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:03.227459    5875 scope.go:122] "RemoveContainer" containerID="9c7e98a17a103e727d7c93c08e8722f96709283a68b409105a3e259fffc7d3d1"
Nov 02 02:15:03.227727 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:15:03.227707    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb_odh-ai-gateway-infra(31bbc102-ee95-4c55-b953-c41f00fd4831)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb" podUID="31bbc102-ee95-4c55-b953-c41f00fd4831"
Nov 02 02:15:03.237196 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:03.237180    5875 scope.go:122] "RemoveContainer" containerID="2277119685c2a29e4c02eadabca6d4b6da1718816103f8f86f4b0fab8be0aeba"
Nov 02 02:15:04.235792 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:04.235761    5875 scope.go:122] "RemoveContainer" containerID="764e843bb30f4189a81a61fb70d867b5b6844a56f53d11dcd1477e2742bab0af"
Nov 02 02:15:04.236258 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:15:04.236079    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs_odh-ai-gateway-infra(44c159c0-914d-49b3-a501-bb0cdfc31e9d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs" podUID="44c159c0-914d-49b3-a501-bb0cdfc31e9d"
Nov 02 02:15:04.237394 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:04.237374    5875 scope.go:122] "RemoveContainer" containerID="9c7e98a17a103e727d7c93c08e8722f96709283a68b409105a3e259fffc7d3d1"
Nov 02 02:15:04.237622 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:15:04.237602    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb_odh-ai-gateway-infra(31bbc102-ee95-4c55-b953-c41f00fd4831)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb" podUID="31bbc102-ee95-4c55-b953-c41f00fd4831"
Nov 02 02:15:09.757006 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:09.756939    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822"
Nov 02 02:15:09.832018 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:09.831990    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90c6117a-dbee-4707-ba61-2e6971c1d52f-kube-api-access-ztvnp\" (UniqueName: \"kubernetes.io/projected/90c6117a-dbee-4707-ba61-2e6971c1d52f-kube-api-access-ztvnp\") pod \"90c6117a-dbee-4707-ba61-2e6971c1d52f\" (UID: \"90c6117a-dbee-4707-ba61-2e6971c1d52f\") "
Nov 02 02:15:09.832178 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:09.832096    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90c6117a-dbee-4707-ba61-2e6971c1d52f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c6117a-dbee-4707-ba61-2e6971c1d52f-plugins-config-volume\") pod \"90c6117a-dbee-4707-ba61-2e6971c1d52f\" (UID: \"90c6117a-dbee-4707-ba61-2e6971c1d52f\") "
Nov 02 02:15:09.832554 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:09.832522    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c6117a-dbee-4707-ba61-2e6971c1d52f-plugins-config-volume" pod "90c6117a-dbee-4707-ba61-2e6971c1d52f" (UID: "90c6117a-dbee-4707-ba61-2e6971c1d52f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:15:09.834233 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:09.834152    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c6117a-dbee-4707-ba61-2e6971c1d52f-kube-api-access-ztvnp" pod "90c6117a-dbee-4707-ba61-2e6971c1d52f" (UID: "90c6117a-dbee-4707-ba61-2e6971c1d52f"). InnerVolumeSpecName "kube-api-access-ztvnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:15:09.845134 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:09.845111    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr"
Nov 02 02:15:09.932882 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:09.932851    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70879f63-ab1e-41ba-83e8-4df35dc40d1d-kube-api-access-w9g4k\" (UniqueName: \"kubernetes.io/projected/70879f63-ab1e-41ba-83e8-4df35dc40d1d-kube-api-access-w9g4k\") pod \"70879f63-ab1e-41ba-83e8-4df35dc40d1d\" (UID: \"70879f63-ab1e-41ba-83e8-4df35dc40d1d\") "
Nov 02 02:15:09.933056 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:09.932924    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70879f63-ab1e-41ba-83e8-4df35dc40d1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70879f63-ab1e-41ba-83e8-4df35dc40d1d-plugins-config-volume\") pod \"70879f63-ab1e-41ba-83e8-4df35dc40d1d\" (UID: \"70879f63-ab1e-41ba-83e8-4df35dc40d1d\") "
Nov 02 02:15:09.933183 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:09.933170    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c6117a-dbee-4707-ba61-2e6971c1d52f-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:09.933253 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:09.933189    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztvnp\" (UniqueName: \"kubernetes.io/projected/90c6117a-dbee-4707-ba61-2e6971c1d52f-kube-api-access-ztvnp\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:09.933307 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:09.933282    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70879f63-ab1e-41ba-83e8-4df35dc40d1d-plugins-config-volume" pod "70879f63-ab1e-41ba-83e8-4df35dc40d1d" (UID: "70879f63-ab1e-41ba-83e8-4df35dc40d1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:15:09.935077 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:09.935057    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70879f63-ab1e-41ba-83e8-4df35dc40d1d-kube-api-access-w9g4k" pod "70879f63-ab1e-41ba-83e8-4df35dc40d1d" (UID: "70879f63-ab1e-41ba-83e8-4df35dc40d1d"). InnerVolumeSpecName "kube-api-access-w9g4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:15:10.034130 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:10.034053    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9g4k\" (UniqueName: \"kubernetes.io/projected/70879f63-ab1e-41ba-83e8-4df35dc40d1d-kube-api-access-w9g4k\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:10.034130 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:10.034083    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70879f63-ab1e-41ba-83e8-4df35dc40d1d-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:10.276846 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:10.276809    5875 generic.go:356] "Generic (PLEG): container finished" podID="90c6117a-dbee-4707-ba61-2e6971c1d52f" containerID="7afe8cd641902e780ff2e240c8e0d2e12b12d59bb147494b10618c3bda4f9ecb" exitCode=0
Nov 02 02:15:10.277035 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:10.276962    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822" event={"ID":"90c6117a-dbee-4707-ba61-2e6971c1d52f","Type":"ContainerDied","Data":"7afe8cd641902e780ff2e240c8e0d2e12b12d59bb147494b10618c3bda4f9ecb"}
Nov 02 02:15:10.277084 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:10.277040    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822" event={"ID":"90c6117a-dbee-4707-ba61-2e6971c1d52f","Type":"ContainerDied","Data":"1714c83b7d23ebc631ff430ffaa14743cc42ae3732836e7bfec1335b3a3cd382"}
Nov 02 02:15:10.277084 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:10.277070    5875 scope.go:122] "RemoveContainer" containerID="7afe8cd641902e780ff2e240c8e0d2e12b12d59bb147494b10618c3bda4f9ecb"
Nov 02 02:15:10.277272 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:10.277253    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822"
Nov 02 02:15:10.280750 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:10.280717    5875 generic.go:356] "Generic (PLEG): container finished" podID="70879f63-ab1e-41ba-83e8-4df35dc40d1d" containerID="97c2c7497e33e3cf304f688087cc7e18c1005318b4710764e956e215a399a8ac" exitCode=0
Nov 02 02:15:10.280876 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:10.280767    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr" event={"ID":"70879f63-ab1e-41ba-83e8-4df35dc40d1d","Type":"ContainerDied","Data":"97c2c7497e33e3cf304f688087cc7e18c1005318b4710764e956e215a399a8ac"}
Nov 02 02:15:10.280876 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:10.280813    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr" event={"ID":"70879f63-ab1e-41ba-83e8-4df35dc40d1d","Type":"ContainerDied","Data":"bc256ee29d8a2a559bfe70e9128c19a53ad048272f81b540ec0d152f24f776f3"}
Nov 02 02:15:10.280970 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:10.280898    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr"
Nov 02 02:15:10.291433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:10.290603    5875 scope.go:122] "RemoveContainer" containerID="7afe8cd641902e780ff2e240c8e0d2e12b12d59bb147494b10618c3bda4f9ecb"
Nov 02 02:15:10.291433 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:15:10.291030    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7afe8cd641902e780ff2e240c8e0d2e12b12d59bb147494b10618c3bda4f9ecb\": container with ID starting with 7afe8cd641902e780ff2e240c8e0d2e12b12d59bb147494b10618c3bda4f9ecb not found: ID does not exist" containerID="7afe8cd641902e780ff2e240c8e0d2e12b12d59bb147494b10618c3bda4f9ecb"
Nov 02 02:15:10.291433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:10.291068    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7afe8cd641902e780ff2e240c8e0d2e12b12d59bb147494b10618c3bda4f9ecb"} err="failed to get container status \"7afe8cd641902e780ff2e240c8e0d2e12b12d59bb147494b10618c3bda4f9ecb\": rpc error: code = NotFound desc = could not find container \"7afe8cd641902e780ff2e240c8e0d2e12b12d59bb147494b10618c3bda4f9ecb\": container with ID starting with 7afe8cd641902e780ff2e240c8e0d2e12b12d59bb147494b10618c3bda4f9ecb not found: ID does not exist"
Nov 02 02:15:10.291433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:10.291092    5875 scope.go:122] "RemoveContainer" containerID="97c2c7497e33e3cf304f688087cc7e18c1005318b4710764e956e215a399a8ac"
Nov 02 02:15:10.305757 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:10.305736    5875 scope.go:122] "RemoveContainer" containerID="97c2c7497e33e3cf304f688087cc7e18c1005318b4710764e956e215a399a8ac"
Nov 02 02:15:10.306113 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:15:10.306088    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97c2c7497e33e3cf304f688087cc7e18c1005318b4710764e956e215a399a8ac\": container with ID starting with 97c2c7497e33e3cf304f688087cc7e18c1005318b4710764e956e215a399a8ac not found: ID does not exist" containerID="97c2c7497e33e3cf304f688087cc7e18c1005318b4710764e956e215a399a8ac"
Nov 02 02:15:10.306201 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:10.306124    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c2c7497e33e3cf304f688087cc7e18c1005318b4710764e956e215a399a8ac"} err="failed to get container status \"97c2c7497e33e3cf304f688087cc7e18c1005318b4710764e956e215a399a8ac\": rpc error: code = NotFound desc = could not find container \"97c2c7497e33e3cf304f688087cc7e18c1005318b4710764e956e215a399a8ac\": container with ID starting with 97c2c7497e33e3cf304f688087cc7e18c1005318b4710764e956e215a399a8ac not found: ID does not exist"
Nov 02 02:15:10.442899 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:10.442851    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr"]
Nov 02 02:15:10.447305 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:10.447275    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c62671-84d4d6cd67-gdvnr"]
Nov 02 02:15:10.558007 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:10.557441    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822"]
Nov 02 02:15:10.561830 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:10.561796    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c62671-656c54b597-zs822"]
Nov 02 02:15:10.693114 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:10.693085    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70879f63-ab1e-41ba-83e8-4df35dc40d1d" path="/var/lib/kubelet/pods/70879f63-ab1e-41ba-83e8-4df35dc40d1d/volumes"
Nov 02 02:15:10.693510 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:10.693496    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90c6117a-dbee-4707-ba61-2e6971c1d52f" path="/var/lib/kubelet/pods/90c6117a-dbee-4707-ba61-2e6971c1d52f/volumes"
Nov 02 02:15:11.318462 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:15:11.318425    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:15:14.687032 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:14.686997    5875 scope.go:122] "RemoveContainer" containerID="764e843bb30f4189a81a61fb70d867b5b6844a56f53d11dcd1477e2742bab0af"
Nov 02 02:15:15.324879 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:15.324791    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs" event={"ID":"44c159c0-914d-49b3-a501-bb0cdfc31e9d","Type":"ContainerStarted","Data":"321b559305c907d785bfefe677da72552d1c409b0645e641c53c3d2e0b827c7f"}
Nov 02 02:15:15.687676 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:15.687644    5875 scope.go:122] "RemoveContainer" containerID="9c7e98a17a103e727d7c93c08e8722f96709283a68b409105a3e259fffc7d3d1"
Nov 02 02:15:16.001450 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:16.001423    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:15:16.333858 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:16.333822    5875 generic.go:356] "Generic (PLEG): container finished" podID="44c159c0-914d-49b3-a501-bb0cdfc31e9d" containerID="321b559305c907d785bfefe677da72552d1c409b0645e641c53c3d2e0b827c7f" exitCode=60
Nov 02 02:15:16.334067 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:16.333881    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs" event={"ID":"44c159c0-914d-49b3-a501-bb0cdfc31e9d","Type":"ContainerDied","Data":"321b559305c907d785bfefe677da72552d1c409b0645e641c53c3d2e0b827c7f"}
Nov 02 02:15:16.334067 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:16.333941    5875 scope.go:122] "RemoveContainer" containerID="764e843bb30f4189a81a61fb70d867b5b6844a56f53d11dcd1477e2742bab0af"
Nov 02 02:15:16.334388 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:16.334369    5875 scope.go:122] "RemoveContainer" containerID="321b559305c907d785bfefe677da72552d1c409b0645e641c53c3d2e0b827c7f"
Nov 02 02:15:16.334687 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:15:16.334666    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs_odh-ai-gateway-infra(44c159c0-914d-49b3-a501-bb0cdfc31e9d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs" podUID="44c159c0-914d-49b3-a501-bb0cdfc31e9d"
Nov 02 02:15:16.336671 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:16.336650    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb" event={"ID":"31bbc102-ee95-4c55-b953-c41f00fd4831","Type":"ContainerStarted","Data":"6b7e51ee1185c304a71823333e3308c9c40d396a1c90b19a461b932915888088"}
Nov 02 02:15:16.461511 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:16.461478    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs"]
Nov 02 02:15:16.566025 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:16.565965    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb" podStartSLOduration=16.565950082 podStartE2EDuration="16.565950082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:15:16.56283576 +0000 UTC m=+2564.003872295" watchObservedRunningTime="2026-11-02 02:15:16.565950082 +0000 UTC m=+2564.006986579"
Nov 02 02:15:17.354090 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:17.354058    5875 generic.go:356] "Generic (PLEG): container finished" podID="31bbc102-ee95-4c55-b953-c41f00fd4831" containerID="6b7e51ee1185c304a71823333e3308c9c40d396a1c90b19a461b932915888088" exitCode=60
Nov 02 02:15:17.354497 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:17.354119    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb" event={"ID":"31bbc102-ee95-4c55-b953-c41f00fd4831","Type":"ContainerDied","Data":"6b7e51ee1185c304a71823333e3308c9c40d396a1c90b19a461b932915888088"}
Nov 02 02:15:17.354497 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:17.354186    5875 scope.go:122] "RemoveContainer" containerID="9c7e98a17a103e727d7c93c08e8722f96709283a68b409105a3e259fffc7d3d1"
Nov 02 02:15:17.354597 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:17.354567    5875 scope.go:122] "RemoveContainer" containerID="6b7e51ee1185c304a71823333e3308c9c40d396a1c90b19a461b932915888088"
Nov 02 02:15:17.354861 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:15:17.354841    5875 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb_odh-ai-gateway-infra(31bbc102-ee95-4c55-b953-c41f00fd4831)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb" podUID="31bbc102-ee95-4c55-b953-c41f00fd4831"
Nov 02 02:15:17.508027 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:17.507999    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs"
Nov 02 02:15:17.575830 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:17.573996    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb"]
Nov 02 02:15:17.621630 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:17.621536    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44c159c0-914d-49b3-a501-bb0cdfc31e9d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44c159c0-914d-49b3-a501-bb0cdfc31e9d-maas-api-service-ca\") pod \"44c159c0-914d-49b3-a501-bb0cdfc31e9d\" (UID: \"44c159c0-914d-49b3-a501-bb0cdfc31e9d\") "
Nov 02 02:15:17.621817 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:17.621715    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44c159c0-914d-49b3-a501-bb0cdfc31e9d-kube-api-access-5jlvd\" (UniqueName: \"kubernetes.io/projected/44c159c0-914d-49b3-a501-bb0cdfc31e9d-kube-api-access-5jlvd\") pod \"44c159c0-914d-49b3-a501-bb0cdfc31e9d\" (UID: \"44c159c0-914d-49b3-a501-bb0cdfc31e9d\") "
Nov 02 02:15:17.622038 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:17.621996    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44c159c0-914d-49b3-a501-bb0cdfc31e9d-maas-api-service-ca" pod "44c159c0-914d-49b3-a501-bb0cdfc31e9d" (UID: "44c159c0-914d-49b3-a501-bb0cdfc31e9d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:15:17.625730 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:17.625686    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c159c0-914d-49b3-a501-bb0cdfc31e9d-kube-api-access-5jlvd" pod "44c159c0-914d-49b3-a501-bb0cdfc31e9d" (UID: "44c159c0-914d-49b3-a501-bb0cdfc31e9d"). InnerVolumeSpecName "kube-api-access-5jlvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:15:17.723244 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:17.723044    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jlvd\" (UniqueName: \"kubernetes.io/projected/44c159c0-914d-49b3-a501-bb0cdfc31e9d-kube-api-access-5jlvd\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:17.723244 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:17.723083    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44c159c0-914d-49b3-a501-bb0cdfc31e9d-maas-api-service-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:18.363518 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:18.363495    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs"
Nov 02 02:15:18.363898 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:18.363490    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs" event={"ID":"44c159c0-914d-49b3-a501-bb0cdfc31e9d","Type":"ContainerDied","Data":"6c9ad30c2399e89dba86ece1b0b313b50c146727dc7202763fda0af814e6c585"}
Nov 02 02:15:18.363898 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:18.363649    5875 scope.go:122] "RemoveContainer" containerID="321b559305c907d785bfefe677da72552d1c409b0645e641c53c3d2e0b827c7f"
Nov 02 02:15:18.551817 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:18.551794    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb"
Nov 02 02:15:18.590689 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:18.590658    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs"]
Nov 02 02:15:18.597061 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:18.597036    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-0ebdb3-29893095-9x9zs"]
Nov 02 02:15:18.633021 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:18.632949    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31bbc102-ee95-4c55-b953-c41f00fd4831-kube-api-access-rs2pj\" (UniqueName: \"kubernetes.io/projected/31bbc102-ee95-4c55-b953-c41f00fd4831-kube-api-access-rs2pj\") pod \"31bbc102-ee95-4c55-b953-c41f00fd4831\" (UID: \"31bbc102-ee95-4c55-b953-c41f00fd4831\") "
Nov 02 02:15:18.633159 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:18.633071    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31bbc102-ee95-4c55-b953-c41f00fd4831-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31bbc102-ee95-4c55-b953-c41f00fd4831-maas-api-service-ca\") pod \"31bbc102-ee95-4c55-b953-c41f00fd4831\" (UID: \"31bbc102-ee95-4c55-b953-c41f00fd4831\") "
Nov 02 02:15:18.633447 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:18.633426    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31bbc102-ee95-4c55-b953-c41f00fd4831-maas-api-service-ca" pod "31bbc102-ee95-4c55-b953-c41f00fd4831" (UID: "31bbc102-ee95-4c55-b953-c41f00fd4831"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:15:18.635334 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:18.635309    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31bbc102-ee95-4c55-b953-c41f00fd4831-kube-api-access-rs2pj" pod "31bbc102-ee95-4c55-b953-c41f00fd4831" (UID: "31bbc102-ee95-4c55-b953-c41f00fd4831"). InnerVolumeSpecName "kube-api-access-rs2pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:15:18.694092 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:18.694056    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44c159c0-914d-49b3-a501-bb0cdfc31e9d" path="/var/lib/kubelet/pods/44c159c0-914d-49b3-a501-bb0cdfc31e9d/volumes"
Nov 02 02:15:18.734842 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:18.734808    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31bbc102-ee95-4c55-b953-c41f00fd4831-maas-api-service-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:18.734842 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:18.734835    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs2pj\" (UniqueName: \"kubernetes.io/projected/31bbc102-ee95-4c55-b953-c41f00fd4831-kube-api-access-rs2pj\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:19.372043 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:19.372015    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb"
Nov 02 02:15:19.372528 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:19.372051    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb" event={"ID":"31bbc102-ee95-4c55-b953-c41f00fd4831","Type":"ContainerDied","Data":"8a47e715e107d5357f1f7d20cd7d832cae680539cc0ec3959f862831acacecae"}
Nov 02 02:15:19.372528 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:19.372131    5875 scope.go:122] "RemoveContainer" containerID="6b7e51ee1185c304a71823333e3308c9c40d396a1c90b19a461b932915888088"
Nov 02 02:15:19.537053 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:19.537013    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb"]
Nov 02 02:15:19.586413 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:19.586381    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c62671-29893095-gmztb"]
Nov 02 02:15:20.693623 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:20.693588    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31bbc102-ee95-4c55-b953-c41f00fd4831" path="/var/lib/kubelet/pods/31bbc102-ee95-4c55-b953-c41f00fd4831/volumes"
Nov 02 02:15:24.511415 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:24.511378    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr"]
Nov 02 02:15:24.560765 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:24.560729    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c62671-8tkdr"]
Nov 02 02:15:24.694519 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:24.694483    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80cb04fe-2a4f-4704-b60b-e0b524f4c2ab" path="/var/lib/kubelet/pods/80cb04fe-2a4f-4704-b60b-e0b524f4c2ab/volumes"
Nov 02 02:15:26.274743 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.274710    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1a38c0d-2bpr7"]
Nov 02 02:15:26.275261 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.275237    5875 cpu_manager.go:414] "containerMap: removing container" podUID="31bbc102-ee95-4c55-b953-c41f00fd4831" containerName="cleanup"
Nov 02 02:15:26.275261 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.275256    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31bbc102-ee95-4c55-b953-c41f00fd4831" containerName="cleanup"
Nov 02 02:15:26.275421 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.275271    5875 cpu_manager.go:414] "containerMap: removing container" podUID="49894b1d-dea5-4978-b51f-b3e6f80b1326" containerName="revoke-keys"
Nov 02 02:15:26.275421 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.275277    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49894b1d-dea5-4978-b51f-b3e6f80b1326" containerName="revoke-keys"
Nov 02 02:15:26.275421 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.275293    5875 cpu_manager.go:414] "containerMap: removing container" podUID="70879f63-ab1e-41ba-83e8-4df35dc40d1d" containerName="payload-pre-processing"
Nov 02 02:15:26.275421 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.275299    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70879f63-ab1e-41ba-83e8-4df35dc40d1d" containerName="payload-pre-processing"
Nov 02 02:15:26.275421 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.275309    5875 cpu_manager.go:414] "containerMap: removing container" podUID="44c159c0-914d-49b3-a501-bb0cdfc31e9d" containerName="cleanup"
Nov 02 02:15:26.275421 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.275313    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c159c0-914d-49b3-a501-bb0cdfc31e9d" containerName="cleanup"
Nov 02 02:15:26.275421 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.275324    5875 cpu_manager.go:414] "containerMap: removing container" podUID="31bbc102-ee95-4c55-b953-c41f00fd4831" containerName="cleanup"
Nov 02 02:15:26.275421 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.275330    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31bbc102-ee95-4c55-b953-c41f00fd4831" containerName="cleanup"
Nov 02 02:15:26.275421 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.275335    5875 cpu_manager.go:414] "containerMap: removing container" podUID="44c159c0-914d-49b3-a501-bb0cdfc31e9d" containerName="cleanup"
Nov 02 02:15:26.275421 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.275339    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c159c0-914d-49b3-a501-bb0cdfc31e9d" containerName="cleanup"
Nov 02 02:15:26.275421 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.275348    5875 cpu_manager.go:414] "containerMap: removing container" podUID="44c159c0-914d-49b3-a501-bb0cdfc31e9d" containerName="cleanup"
Nov 02 02:15:26.275421 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.275352    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c159c0-914d-49b3-a501-bb0cdfc31e9d" containerName="cleanup"
Nov 02 02:15:26.275421 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.275363    5875 cpu_manager.go:414] "containerMap: removing container" podUID="90c6117a-dbee-4707-ba61-2e6971c1d52f" containerName="payload-processing"
Nov 02 02:15:26.275421 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.275367    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90c6117a-dbee-4707-ba61-2e6971c1d52f" containerName="payload-processing"
Nov 02 02:15:26.275854 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.275452    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="31bbc102-ee95-4c55-b953-c41f00fd4831" containerName="cleanup"
Nov 02 02:15:26.275854 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.275460    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="49894b1d-dea5-4978-b51f-b3e6f80b1326" containerName="revoke-keys"
Nov 02 02:15:26.275854 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.275468    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="70879f63-ab1e-41ba-83e8-4df35dc40d1d" containerName="payload-pre-processing"
Nov 02 02:15:26.275854 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.275475    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="44c159c0-914d-49b3-a501-bb0cdfc31e9d" containerName="cleanup"
Nov 02 02:15:26.275854 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.275481    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="31bbc102-ee95-4c55-b953-c41f00fd4831" containerName="cleanup"
Nov 02 02:15:26.275854 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.275487    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="44c159c0-914d-49b3-a501-bb0cdfc31e9d" containerName="cleanup"
Nov 02 02:15:26.275854 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.275493    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="31bbc102-ee95-4c55-b953-c41f00fd4831" containerName="cleanup"
Nov 02 02:15:26.275854 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.275499    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="90c6117a-dbee-4707-ba61-2e6971c1d52f" containerName="payload-processing"
Nov 02 02:15:26.278732 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.278717    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1a38c0d-2bpr7"
Nov 02 02:15:26.336760 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.336726    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1a38c0d-2bpr7"]
Nov 02 02:15:26.401740 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.401705    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:15:26.401936 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.401888    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:15:26.427370 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.427343    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxqkk\" (UniqueName: \"kubernetes.io/projected/7e9e1f5e-d36d-4e4e-a990-f9da53c59187-kube-api-access-qxqkk\") pod \"maas-api-revoke-sub-c1a38c0d-2bpr7\" (UID: \"7e9e1f5e-d36d-4e4e-a990-f9da53c59187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1a38c0d-2bpr7"
Nov 02 02:15:26.427531 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.427386    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e9e1f5e-d36d-4e4e-a990-f9da53c59187-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1a38c0d-2bpr7\" (UID: \"7e9e1f5e-d36d-4e4e-a990-f9da53c59187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1a38c0d-2bpr7"
Nov 02 02:15:26.427531 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.427481    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e9e1f5e-d36d-4e4e-a990-f9da53c59187-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1a38c0d-2bpr7\" (UID: \"7e9e1f5e-d36d-4e4e-a990-f9da53c59187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1a38c0d-2bpr7"
Nov 02 02:15:26.529577 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.529479    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxqkk\" (UniqueName: \"kubernetes.io/projected/7e9e1f5e-d36d-4e4e-a990-f9da53c59187-kube-api-access-qxqkk\") pod \"maas-api-revoke-sub-c1a38c0d-2bpr7\" (UID: \"7e9e1f5e-d36d-4e4e-a990-f9da53c59187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1a38c0d-2bpr7"
Nov 02 02:15:26.529577 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.529552    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e9e1f5e-d36d-4e4e-a990-f9da53c59187-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1a38c0d-2bpr7\" (UID: \"7e9e1f5e-d36d-4e4e-a990-f9da53c59187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1a38c0d-2bpr7"
Nov 02 02:15:26.529831 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.529611    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e9e1f5e-d36d-4e4e-a990-f9da53c59187-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1a38c0d-2bpr7\" (UID: \"7e9e1f5e-d36d-4e4e-a990-f9da53c59187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1a38c0d-2bpr7"
Nov 02 02:15:26.530361 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.530339    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e9e1f5e-d36d-4e4e-a990-f9da53c59187-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1a38c0d-2bpr7\" (UID: \"7e9e1f5e-d36d-4e4e-a990-f9da53c59187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1a38c0d-2bpr7"
Nov 02 02:15:26.541311 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.532039    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e9e1f5e-d36d-4e4e-a990-f9da53c59187-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1a38c0d-2bpr7\" (UID: \"7e9e1f5e-d36d-4e4e-a990-f9da53c59187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1a38c0d-2bpr7"
Nov 02 02:15:26.590618 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.590584    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxqkk\" (UniqueName: \"kubernetes.io/projected/7e9e1f5e-d36d-4e4e-a990-f9da53c59187-kube-api-access-qxqkk\") pod \"maas-api-revoke-sub-c1a38c0d-2bpr7\" (UID: \"7e9e1f5e-d36d-4e4e-a990-f9da53c59187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1a38c0d-2bpr7"
Nov 02 02:15:26.592453 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.592431    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1a38c0d-2bpr7"
Nov 02 02:15:26.996644 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:26.996618    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1a38c0d-2bpr7"]
Nov 02 02:15:26.998421 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:15:26.998388    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9e1f5e_d36d_4e4e_a990_f9da53c59187.slice/crio-d538c5ded3731bc44d0fcbfcf0d82f435c393fc6be99fe99a578abb8a7a68183 WatchSource:0}: Error finding container d538c5ded3731bc44d0fcbfcf0d82f435c393fc6be99fe99a578abb8a7a68183: Status 404 returned error can't find the container with id d538c5ded3731bc44d0fcbfcf0d82f435c393fc6be99fe99a578abb8a7a68183
Nov 02 02:15:27.420382 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:27.420335    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1a38c0d-2bpr7" event={"ID":"7e9e1f5e-d36d-4e4e-a990-f9da53c59187","Type":"ContainerStarted","Data":"41285e3c052d4b9de3d2d6837b2891816efffe3492561fbfab2677fcd3f5e826"}
Nov 02 02:15:27.420829 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:27.420393    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1a38c0d-2bpr7" event={"ID":"7e9e1f5e-d36d-4e4e-a990-f9da53c59187","Type":"ContainerStarted","Data":"d538c5ded3731bc44d0fcbfcf0d82f435c393fc6be99fe99a578abb8a7a68183"}
Nov 02 02:15:27.535412 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:27.535357    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1a38c0d-2bpr7" podStartSLOduration=1.535341157 podStartE2EDuration="1.535341157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:15:27.534490128 +0000 UTC m=+2574.975526625" watchObservedRunningTime="2026-11-02 02:15:27.535341157 +0000 UTC m=+2574.976377694"
Nov 02 02:15:28.428643 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:28.428613    5875 generic.go:356] "Generic (PLEG): container finished" podID="7e9e1f5e-d36d-4e4e-a990-f9da53c59187" containerID="41285e3c052d4b9de3d2d6837b2891816efffe3492561fbfab2677fcd3f5e826" exitCode=0
Nov 02 02:15:28.429047 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:28.428703    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1a38c0d-2bpr7" event={"ID":"7e9e1f5e-d36d-4e4e-a990-f9da53c59187","Type":"ContainerDied","Data":"41285e3c052d4b9de3d2d6837b2891816efffe3492561fbfab2677fcd3f5e826"}
Nov 02 02:15:29.576964 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:29.576938    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1a38c0d-2bpr7"
Nov 02 02:15:29.677445 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:29.677415    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e9e1f5e-d36d-4e4e-a990-f9da53c59187-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e9e1f5e-d36d-4e4e-a990-f9da53c59187-maas-api-xks-ca\") pod \"7e9e1f5e-d36d-4e4e-a990-f9da53c59187\" (UID: \"7e9e1f5e-d36d-4e4e-a990-f9da53c59187\") "
Nov 02 02:15:29.677636 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:29.677466    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e9e1f5e-d36d-4e4e-a990-f9da53c59187-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e9e1f5e-d36d-4e4e-a990-f9da53c59187-maas-api-openshift-ca\") pod \"7e9e1f5e-d36d-4e4e-a990-f9da53c59187\" (UID: \"7e9e1f5e-d36d-4e4e-a990-f9da53c59187\") "
Nov 02 02:15:29.677636 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:29.677612    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e9e1f5e-d36d-4e4e-a990-f9da53c59187-kube-api-access-qxqkk\" (UniqueName: \"kubernetes.io/projected/7e9e1f5e-d36d-4e4e-a990-f9da53c59187-kube-api-access-qxqkk\") pod \"7e9e1f5e-d36d-4e4e-a990-f9da53c59187\" (UID: \"7e9e1f5e-d36d-4e4e-a990-f9da53c59187\") "
Nov 02 02:15:29.677833 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:29.677812    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e9e1f5e-d36d-4e4e-a990-f9da53c59187-maas-api-openshift-ca" pod "7e9e1f5e-d36d-4e4e-a990-f9da53c59187" (UID: "7e9e1f5e-d36d-4e4e-a990-f9da53c59187"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:15:29.679718 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:29.679690    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e9e1f5e-d36d-4e4e-a990-f9da53c59187-maas-api-xks-ca" pod "7e9e1f5e-d36d-4e4e-a990-f9da53c59187" (UID: "7e9e1f5e-d36d-4e4e-a990-f9da53c59187"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:15:29.679864 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:29.679838    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9e1f5e-d36d-4e4e-a990-f9da53c59187-kube-api-access-qxqkk" pod "7e9e1f5e-d36d-4e4e-a990-f9da53c59187" (UID: "7e9e1f5e-d36d-4e4e-a990-f9da53c59187"). InnerVolumeSpecName "kube-api-access-qxqkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:15:29.779096 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:29.779057    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e9e1f5e-d36d-4e4e-a990-f9da53c59187-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:29.779096 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:29.779090    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e9e1f5e-d36d-4e4e-a990-f9da53c59187-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:29.779096 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:29.779101    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxqkk\" (UniqueName: \"kubernetes.io/projected/7e9e1f5e-d36d-4e4e-a990-f9da53c59187-kube-api-access-qxqkk\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:30.446517 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:30.446488    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1a38c0d-2bpr7"
Nov 02 02:15:30.446699 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:30.446526    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1a38c0d-2bpr7" event={"ID":"7e9e1f5e-d36d-4e4e-a990-f9da53c59187","Type":"ContainerDied","Data":"d538c5ded3731bc44d0fcbfcf0d82f435c393fc6be99fe99a578abb8a7a68183"}
Nov 02 02:15:30.446699 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:30.446575    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d538c5ded3731bc44d0fcbfcf0d82f435c393fc6be99fe99a578abb8a7a68183"
Nov 02 02:15:39.040547 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:39.040514    5875 scope.go:122] "RemoveContainer" containerID="b7d24dd00ab46d819cb67eafd5bb141c4d248c282f134ce6588acfe7dcf3c039"
Nov 02 02:15:41.318753 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:15:41.318709    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:15:41.625310 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:41.625230    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87e1e297-6t6hr"]
Nov 02 02:15:41.625717 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:41.625704    5875 cpu_manager.go:414] "containerMap: removing container" podUID="31bbc102-ee95-4c55-b953-c41f00fd4831" containerName="cleanup"
Nov 02 02:15:41.625780 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:41.625718    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31bbc102-ee95-4c55-b953-c41f00fd4831" containerName="cleanup"
Nov 02 02:15:41.625780 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:41.625754    5875 cpu_manager.go:414] "containerMap: removing container" podUID="7e9e1f5e-d36d-4e4e-a990-f9da53c59187" containerName="revoke-keys"
Nov 02 02:15:41.625780 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:41.625759    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e9e1f5e-d36d-4e4e-a990-f9da53c59187" containerName="revoke-keys"
Nov 02 02:15:41.625931 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:41.625825    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e9e1f5e-d36d-4e4e-a990-f9da53c59187" containerName="revoke-keys"
Nov 02 02:15:41.625931 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:41.625837    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="44c159c0-914d-49b3-a501-bb0cdfc31e9d" containerName="cleanup"
Nov 02 02:15:41.630633 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:41.630617    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e1e297-6t6hr"
Nov 02 02:15:41.683960 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:41.683912    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:15:41.702120 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:41.702074    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69m4r\" (UniqueName: \"kubernetes.io/projected/0b9d0472-466e-444a-b2ee-309738376172-kube-api-access-69m4r\") pod \"maas-api-revoke-sub-87e1e297-6t6hr\" (UID: \"0b9d0472-466e-444a-b2ee-309738376172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e1e297-6t6hr"
Nov 02 02:15:41.702342 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:41.702152    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b9d0472-466e-444a-b2ee-309738376172-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87e1e297-6t6hr\" (UID: \"0b9d0472-466e-444a-b2ee-309738376172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e1e297-6t6hr"
Nov 02 02:15:41.702342 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:41.702208    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b9d0472-466e-444a-b2ee-309738376172-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87e1e297-6t6hr\" (UID: \"0b9d0472-466e-444a-b2ee-309738376172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e1e297-6t6hr"
Nov 02 02:15:41.733292 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:41.733254    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:15:41.737995 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:41.737964    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87e1e297-6t6hr"]
Nov 02 02:15:41.803395 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:41.803366    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69m4r\" (UniqueName: \"kubernetes.io/projected/0b9d0472-466e-444a-b2ee-309738376172-kube-api-access-69m4r\") pod \"maas-api-revoke-sub-87e1e297-6t6hr\" (UID: \"0b9d0472-466e-444a-b2ee-309738376172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e1e297-6t6hr"
Nov 02 02:15:41.803576 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:41.803409    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b9d0472-466e-444a-b2ee-309738376172-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87e1e297-6t6hr\" (UID: \"0b9d0472-466e-444a-b2ee-309738376172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e1e297-6t6hr"
Nov 02 02:15:41.803576 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:41.803459    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b9d0472-466e-444a-b2ee-309738376172-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87e1e297-6t6hr\" (UID: \"0b9d0472-466e-444a-b2ee-309738376172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e1e297-6t6hr"
Nov 02 02:15:41.804932 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:41.804888    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b9d0472-466e-444a-b2ee-309738376172-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87e1e297-6t6hr\" (UID: \"0b9d0472-466e-444a-b2ee-309738376172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e1e297-6t6hr"
Nov 02 02:15:41.814617 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:41.814588    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b9d0472-466e-444a-b2ee-309738376172-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87e1e297-6t6hr\" (UID: \"0b9d0472-466e-444a-b2ee-309738376172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e1e297-6t6hr"
Nov 02 02:15:41.865041 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:41.865007    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69m4r\" (UniqueName: \"kubernetes.io/projected/0b9d0472-466e-444a-b2ee-309738376172-kube-api-access-69m4r\") pod \"maas-api-revoke-sub-87e1e297-6t6hr\" (UID: \"0b9d0472-466e-444a-b2ee-309738376172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e1e297-6t6hr"
Nov 02 02:15:41.943443 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:41.943358    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e1e297-6t6hr"
Nov 02 02:15:42.318154 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:42.318125    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87e1e297-6t6hr"]
Nov 02 02:15:42.319627 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:15:42.319599    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b9d0472_466e_444a_b2ee_309738376172.slice/crio-88458a17217778024111792528aae0bc035616821266504f4fcf310926122932 WatchSource:0}: Error finding container 88458a17217778024111792528aae0bc035616821266504f4fcf310926122932: Status 404 returned error can't find the container with id 88458a17217778024111792528aae0bc035616821266504f4fcf310926122932
Nov 02 02:15:42.525689 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:42.525659    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e1e297-6t6hr" event={"ID":"0b9d0472-466e-444a-b2ee-309738376172","Type":"ContainerStarted","Data":"88458a17217778024111792528aae0bc035616821266504f4fcf310926122932"}
Nov 02 02:15:43.532942 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:43.532902    5875 generic.go:356] "Generic (PLEG): container finished" podID="0b9d0472-466e-444a-b2ee-309738376172" containerID="5b03a84ae70dee47db76efef50cfbbf2e5b826e8ad5c3f486125f2d533cdbf74" exitCode=0
Nov 02 02:15:43.533397 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:43.532957    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e1e297-6t6hr" event={"ID":"0b9d0472-466e-444a-b2ee-309738376172","Type":"ContainerDied","Data":"5b03a84ae70dee47db76efef50cfbbf2e5b826e8ad5c3f486125f2d533cdbf74"}
Nov 02 02:15:44.508794 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:44.508760    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:15:44.719373 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:44.719351    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e1e297-6t6hr"
Nov 02 02:15:44.839465 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:44.839387    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b9d0472-466e-444a-b2ee-309738376172-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b9d0472-466e-444a-b2ee-309738376172-maas-api-xks-ca\") pod \"0b9d0472-466e-444a-b2ee-309738376172\" (UID: \"0b9d0472-466e-444a-b2ee-309738376172\") "
Nov 02 02:15:44.839465 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:44.839453    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b9d0472-466e-444a-b2ee-309738376172-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b9d0472-466e-444a-b2ee-309738376172-maas-api-openshift-ca\") pod \"0b9d0472-466e-444a-b2ee-309738376172\" (UID: \"0b9d0472-466e-444a-b2ee-309738376172\") "
Nov 02 02:15:44.839733 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:44.839593    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b9d0472-466e-444a-b2ee-309738376172-kube-api-access-69m4r\" (UniqueName: \"kubernetes.io/projected/0b9d0472-466e-444a-b2ee-309738376172-kube-api-access-69m4r\") pod \"0b9d0472-466e-444a-b2ee-309738376172\" (UID: \"0b9d0472-466e-444a-b2ee-309738376172\") "
Nov 02 02:15:44.839828 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:44.839804    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b9d0472-466e-444a-b2ee-309738376172-maas-api-openshift-ca" pod "0b9d0472-466e-444a-b2ee-309738376172" (UID: "0b9d0472-466e-444a-b2ee-309738376172"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:15:44.839979 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:44.839960    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b9d0472-466e-444a-b2ee-309738376172-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:44.841682 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:44.841651    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b9d0472-466e-444a-b2ee-309738376172-maas-api-xks-ca" pod "0b9d0472-466e-444a-b2ee-309738376172" (UID: "0b9d0472-466e-444a-b2ee-309738376172"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:15:44.841816 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:44.841724    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b9d0472-466e-444a-b2ee-309738376172-kube-api-access-69m4r" pod "0b9d0472-466e-444a-b2ee-309738376172" (UID: "0b9d0472-466e-444a-b2ee-309738376172"). InnerVolumeSpecName "kube-api-access-69m4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:15:44.941057 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:44.941027    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b9d0472-466e-444a-b2ee-309738376172-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:44.941057 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:44.941052    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69m4r\" (UniqueName: \"kubernetes.io/projected/0b9d0472-466e-444a-b2ee-309738376172-kube-api-access-69m4r\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:15:45.544744 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:45.544715    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e1e297-6t6hr"
Nov 02 02:15:45.544963 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:45.544781    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e1e297-6t6hr" event={"ID":"0b9d0472-466e-444a-b2ee-309738376172","Type":"ContainerDied","Data":"88458a17217778024111792528aae0bc035616821266504f4fcf310926122932"}
Nov 02 02:15:45.544963 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:45.544821    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88458a17217778024111792528aae0bc035616821266504f4fcf310926122932"
Nov 02 02:15:48.046351 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:48.046305    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d426fc-jpcw9"]
Nov 02 02:15:48.095757 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:48.095723    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d426fc-jpcw9"]
Nov 02 02:15:48.693447 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:15:48.693414    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9dbc65e-d84e-46e7-be0d-6989c268c09f" path="/var/lib/kubelet/pods/e9dbc65e-d84e-46e7-be0d-6989c268c09f/volumes"
Nov 02 02:16:04.110011 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:04.109976    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:16:08.777694 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:08.777661    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"]
Nov 02 02:16:09.267135 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:09.267105    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg"]
Nov 02 02:16:09.267656 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:09.267642    5875 cpu_manager.go:414] "containerMap: removing container" podUID="0b9d0472-466e-444a-b2ee-309738376172" containerName="revoke-keys"
Nov 02 02:16:09.267714 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:09.267658    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b9d0472-466e-444a-b2ee-309738376172" containerName="revoke-keys"
Nov 02 02:16:09.267788 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:09.267777    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b9d0472-466e-444a-b2ee-309738376172" containerName="revoke-keys"
Nov 02 02:16:09.271294 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:09.271274    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg"
Nov 02 02:16:09.303961 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:09.303931    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17668486-eec6-4551-a99c-7efd65458f0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg\" (UID: \"17668486-eec6-4551-a99c-7efd65458f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg"
Nov 02 02:16:09.304072 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:09.304002    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17668486-eec6-4551-a99c-7efd65458f0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg\" (UID: \"17668486-eec6-4551-a99c-7efd65458f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg"
Nov 02 02:16:09.304072 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:09.304066    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kzgw\" (UniqueName: \"kubernetes.io/projected/17668486-eec6-4551-a99c-7efd65458f0a-kube-api-access-7kzgw\") pod \"maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg\" (UID: \"17668486-eec6-4551-a99c-7efd65458f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg"
Nov 02 02:16:09.325426 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:09.325402    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:16:09.325703 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:09.325670    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:16:09.363080 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:09.363048    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg"]
Nov 02 02:16:09.404711 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:09.404681    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17668486-eec6-4551-a99c-7efd65458f0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg\" (UID: \"17668486-eec6-4551-a99c-7efd65458f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg"
Nov 02 02:16:09.404862 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:09.404729    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17668486-eec6-4551-a99c-7efd65458f0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg\" (UID: \"17668486-eec6-4551-a99c-7efd65458f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg"
Nov 02 02:16:09.404862 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:09.404778    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kzgw\" (UniqueName: \"kubernetes.io/projected/17668486-eec6-4551-a99c-7efd65458f0a-kube-api-access-7kzgw\") pod \"maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg\" (UID: \"17668486-eec6-4551-a99c-7efd65458f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg"
Nov 02 02:16:09.405467 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:09.405445    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17668486-eec6-4551-a99c-7efd65458f0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg\" (UID: \"17668486-eec6-4551-a99c-7efd65458f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg"
Nov 02 02:16:09.407103 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:09.407083    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17668486-eec6-4551-a99c-7efd65458f0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg\" (UID: \"17668486-eec6-4551-a99c-7efd65458f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg"
Nov 02 02:16:09.465631 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:09.465597    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kzgw\" (UniqueName: \"kubernetes.io/projected/17668486-eec6-4551-a99c-7efd65458f0a-kube-api-access-7kzgw\") pod \"maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg\" (UID: \"17668486-eec6-4551-a99c-7efd65458f0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg"
Nov 02 02:16:09.584543 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:09.584483    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg"
Nov 02 02:16:09.956850 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:09.956826    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg"]
Nov 02 02:16:09.958901 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:16:09.958861    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17668486_eec6_4551_a99c_7efd65458f0a.slice/crio-a9336b69d9076dc1195ef501b29c3c16ea74790a0fdfaea8b87da991ba45b0f9 WatchSource:0}: Error finding container a9336b69d9076dc1195ef501b29c3c16ea74790a0fdfaea8b87da991ba45b0f9: Status 404 returned error can't find the container with id a9336b69d9076dc1195ef501b29c3c16ea74790a0fdfaea8b87da991ba45b0f9
Nov 02 02:16:10.709128 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:10.709091    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg" event={"ID":"17668486-eec6-4551-a99c-7efd65458f0a","Type":"ContainerStarted","Data":"5a7728e7a2216aef0f54704f467b702381ecd4c5b5944b21c03250f8905b84c4"}
Nov 02 02:16:10.709311 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:10.709178    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg" event={"ID":"17668486-eec6-4551-a99c-7efd65458f0a","Type":"ContainerStarted","Data":"a9336b69d9076dc1195ef501b29c3c16ea74790a0fdfaea8b87da991ba45b0f9"}
Nov 02 02:16:11.320279 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:16:11.320243    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:16:11.716481 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:11.716436    5875 generic.go:356] "Generic (PLEG): container finished" podID="17668486-eec6-4551-a99c-7efd65458f0a" containerID="5a7728e7a2216aef0f54704f467b702381ecd4c5b5944b21c03250f8905b84c4" exitCode=0
Nov 02 02:16:11.716666 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:11.716519    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg" event={"ID":"17668486-eec6-4551-a99c-7efd65458f0a","Type":"ContainerDied","Data":"5a7728e7a2216aef0f54704f467b702381ecd4c5b5944b21c03250f8905b84c4"}
Nov 02 02:16:12.852807 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:12.852781    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg"
Nov 02 02:16:12.939129 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:12.939095    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17668486-eec6-4551-a99c-7efd65458f0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17668486-eec6-4551-a99c-7efd65458f0a-maas-api-xks-ca\") pod \"17668486-eec6-4551-a99c-7efd65458f0a\" (UID: \"17668486-eec6-4551-a99c-7efd65458f0a\") "
Nov 02 02:16:12.939333 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:12.939140    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17668486-eec6-4551-a99c-7efd65458f0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17668486-eec6-4551-a99c-7efd65458f0a-maas-api-openshift-ca\") pod \"17668486-eec6-4551-a99c-7efd65458f0a\" (UID: \"17668486-eec6-4551-a99c-7efd65458f0a\") "
Nov 02 02:16:12.939333 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:12.939306    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17668486-eec6-4551-a99c-7efd65458f0a-kube-api-access-7kzgw\" (UniqueName: \"kubernetes.io/projected/17668486-eec6-4551-a99c-7efd65458f0a-kube-api-access-7kzgw\") pod \"17668486-eec6-4551-a99c-7efd65458f0a\" (UID: \"17668486-eec6-4551-a99c-7efd65458f0a\") "
Nov 02 02:16:12.939507 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:12.939486    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17668486-eec6-4551-a99c-7efd65458f0a-maas-api-openshift-ca" pod "17668486-eec6-4551-a99c-7efd65458f0a" (UID: "17668486-eec6-4551-a99c-7efd65458f0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:16:12.939626 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:12.939605    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17668486-eec6-4551-a99c-7efd65458f0a-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:16:12.941516 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:12.941491    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17668486-eec6-4551-a99c-7efd65458f0a-maas-api-xks-ca" pod "17668486-eec6-4551-a99c-7efd65458f0a" (UID: "17668486-eec6-4551-a99c-7efd65458f0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:16:12.941621 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:12.941525    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17668486-eec6-4551-a99c-7efd65458f0a-kube-api-access-7kzgw" pod "17668486-eec6-4551-a99c-7efd65458f0a" (UID: "17668486-eec6-4551-a99c-7efd65458f0a"). InnerVolumeSpecName "kube-api-access-7kzgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:16:13.040862 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:13.040790    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kzgw\" (UniqueName: \"kubernetes.io/projected/17668486-eec6-4551-a99c-7efd65458f0a-kube-api-access-7kzgw\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:16:13.040862 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:13.040817    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17668486-eec6-4551-a99c-7efd65458f0a-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:16:13.729971 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:13.729944    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg"
Nov 02 02:16:13.730171 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:13.729986    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg" event={"ID":"17668486-eec6-4551-a99c-7efd65458f0a","Type":"ContainerDied","Data":"a9336b69d9076dc1195ef501b29c3c16ea74790a0fdfaea8b87da991ba45b0f9"}
Nov 02 02:16:13.730171 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:13.730033    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9336b69d9076dc1195ef501b29c3c16ea74790a0fdfaea8b87da991ba45b0f9"
Nov 02 02:16:20.998001 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:20.997963    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:16:23.798208 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:23.798147    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4" podUID="c30b1225-cfec-4f91-9fa0-ab84afcb0598" containerName="main" containerID="cri-o://fda5f7baed2ffdfd1a83105a2a3e29cc14d7c09299ee959bfdd8bccb1b482706" gracePeriod=45
Nov 02 02:16:24.066582 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.066520    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:16:24.175479 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.175439    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c30b1225-cfec-4f91-9fa0-ab84afcb0598-tls-certs\" (UniqueName: \"kubernetes.io/secret/c30b1225-cfec-4f91-9fa0-ab84afcb0598-tls-certs\") pod \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\" (UID: \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\") "
Nov 02 02:16:24.175680 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.175525    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-dshm\") pod \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\" (UID: \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\") "
Nov 02 02:16:24.175680 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.175622    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-tmp-dir\") pod \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\" (UID: \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\") "
Nov 02 02:16:24.175680 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.175673    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-home\" (UniqueName: \"kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-home\") pod \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\" (UID: \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\") "
Nov 02 02:16:24.175809 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.175755    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c30b1225-cfec-4f91-9fa0-ab84afcb0598-kube-api-access-r26tc\" (UniqueName: \"kubernetes.io/projected/c30b1225-cfec-4f91-9fa0-ab84afcb0598-kube-api-access-r26tc\") pod \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\" (UID: \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\") "
Nov 02 02:16:24.175853 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.175811    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-model-cache\") pod \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\" (UID: \"c30b1225-cfec-4f91-9fa0-ab84afcb0598\") "
Nov 02 02:16:24.175935 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.175874    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-tmp-dir" pod "c30b1225-cfec-4f91-9fa0-ab84afcb0598" (UID: "c30b1225-cfec-4f91-9fa0-ab84afcb0598"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:16:24.176002 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.175940    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-home" pod "c30b1225-cfec-4f91-9fa0-ab84afcb0598" (UID: "c30b1225-cfec-4f91-9fa0-ab84afcb0598"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:16:24.176116 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.176093    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-model-cache" pod "c30b1225-cfec-4f91-9fa0-ab84afcb0598" (UID: "c30b1225-cfec-4f91-9fa0-ab84afcb0598"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:16:24.176292 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.176276    5875 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-model-cache\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:16:24.176353 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.176298    5875 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-tmp-dir\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:16:24.176353 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.176313    5875 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-home\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:16:24.178051 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.178029    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c30b1225-cfec-4f91-9fa0-ab84afcb0598-kube-api-access-r26tc" pod "c30b1225-cfec-4f91-9fa0-ab84afcb0598" (UID: "c30b1225-cfec-4f91-9fa0-ab84afcb0598"). InnerVolumeSpecName "kube-api-access-r26tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:16:24.178208 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.178188    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-dshm" pod "c30b1225-cfec-4f91-9fa0-ab84afcb0598" (UID: "c30b1225-cfec-4f91-9fa0-ab84afcb0598"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:16:24.178300 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.178188    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c30b1225-cfec-4f91-9fa0-ab84afcb0598-tls-certs" pod "c30b1225-cfec-4f91-9fa0-ab84afcb0598" (UID: "c30b1225-cfec-4f91-9fa0-ab84afcb0598"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:16:24.277246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.277188    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r26tc\" (UniqueName: \"kubernetes.io/projected/c30b1225-cfec-4f91-9fa0-ab84afcb0598-kube-api-access-r26tc\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:16:24.277246 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.277250    5875 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30b1225-cfec-4f91-9fa0-ab84afcb0598-tls-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:16:24.277459 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.277262    5875 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30b1225-cfec-4f91-9fa0-ab84afcb0598-dshm\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:16:24.795528 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.795497    5875 generic.go:356] "Generic (PLEG): container finished" podID="c30b1225-cfec-4f91-9fa0-ab84afcb0598" containerID="fda5f7baed2ffdfd1a83105a2a3e29cc14d7c09299ee959bfdd8bccb1b482706" exitCode=0
Nov 02 02:16:24.795710 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.795545    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"
Nov 02 02:16:24.795710 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.795588    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4" event={"ID":"c30b1225-cfec-4f91-9fa0-ab84afcb0598","Type":"ContainerDied","Data":"fda5f7baed2ffdfd1a83105a2a3e29cc14d7c09299ee959bfdd8bccb1b482706"}
Nov 02 02:16:24.795710 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.795635    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4" event={"ID":"c30b1225-cfec-4f91-9fa0-ab84afcb0598","Type":"ContainerDied","Data":"9a40ebcccd734e902d31e6ea7fc61a5682f205d0312e3e790b4485042be85359"}
Nov 02 02:16:24.795710 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.795656    5875 scope.go:122] "RemoveContainer" containerID="fda5f7baed2ffdfd1a83105a2a3e29cc14d7c09299ee959bfdd8bccb1b482706"
Nov 02 02:16:24.805617 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.805390    5875 scope.go:122] "RemoveContainer" containerID="fda5f7baed2ffdfd1a83105a2a3e29cc14d7c09299ee959bfdd8bccb1b482706"
Nov 02 02:16:24.805873 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:16:24.805646    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fda5f7baed2ffdfd1a83105a2a3e29cc14d7c09299ee959bfdd8bccb1b482706\": container with ID starting with fda5f7baed2ffdfd1a83105a2a3e29cc14d7c09299ee959bfdd8bccb1b482706 not found: ID does not exist" containerID="fda5f7baed2ffdfd1a83105a2a3e29cc14d7c09299ee959bfdd8bccb1b482706"
Nov 02 02:16:24.805873 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.805680    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fda5f7baed2ffdfd1a83105a2a3e29cc14d7c09299ee959bfdd8bccb1b482706"} err="failed to get container status \"fda5f7baed2ffdfd1a83105a2a3e29cc14d7c09299ee959bfdd8bccb1b482706\": rpc error: code = NotFound desc = could not find container \"fda5f7baed2ffdfd1a83105a2a3e29cc14d7c09299ee959bfdd8bccb1b482706\": container with ID starting with fda5f7baed2ffdfd1a83105a2a3e29cc14d7c09299ee959bfdd8bccb1b482706 not found: ID does not exist"
Nov 02 02:16:24.959059 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:24.959023    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"]
Nov 02 02:16:25.008131 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:25.008100    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4211c0/test-model-4211c0-kserve-687b7f8985-5x9z4"]
Nov 02 02:16:26.693643 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:26.693612    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c30b1225-cfec-4f91-9fa0-ab84afcb0598" path="/var/lib/kubelet/pods/c30b1225-cfec-4f91-9fa0-ab84afcb0598/volumes"
Nov 02 02:16:30.964924 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:30.964889    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd"]
Nov 02 02:16:30.965374 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:30.965088    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd" podUID="ef1f600f-0137-46f5-bcfa-9357925458e4" containerName="maas-api" containerID="cri-o://75e5a5f82f7810e6b3b0f1ac778c4e7c47ae38968dcbd7acbfade1502deeb720" gracePeriod=30
Nov 02 02:16:31.239865 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:31.239839    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd"
Nov 02 02:16:31.360245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:31.354097    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef1f600f-0137-46f5-bcfa-9357925458e4-kube-api-access-mfhnd\" (UniqueName: \"kubernetes.io/projected/ef1f600f-0137-46f5-bcfa-9357925458e4-kube-api-access-mfhnd\") pod \"ef1f600f-0137-46f5-bcfa-9357925458e4\" (UID: \"ef1f600f-0137-46f5-bcfa-9357925458e4\") "
Nov 02 02:16:31.360245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:31.354172    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef1f600f-0137-46f5-bcfa-9357925458e4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef1f600f-0137-46f5-bcfa-9357925458e4-openshift-service-ca\") pod \"ef1f600f-0137-46f5-bcfa-9357925458e4\" (UID: \"ef1f600f-0137-46f5-bcfa-9357925458e4\") "
Nov 02 02:16:31.360245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:31.354212    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef1f600f-0137-46f5-bcfa-9357925458e4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef1f600f-0137-46f5-bcfa-9357925458e4-metrics-certs\") pod \"ef1f600f-0137-46f5-bcfa-9357925458e4\" (UID: \"ef1f600f-0137-46f5-bcfa-9357925458e4\") "
Nov 02 02:16:31.360245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:31.354309    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef1f600f-0137-46f5-bcfa-9357925458e4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef1f600f-0137-46f5-bcfa-9357925458e4-maas-api-tls\") pod \"ef1f600f-0137-46f5-bcfa-9357925458e4\" (UID: \"ef1f600f-0137-46f5-bcfa-9357925458e4\") "
Nov 02 02:16:31.360245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:31.354966    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef1f600f-0137-46f5-bcfa-9357925458e4-openshift-service-ca" pod "ef1f600f-0137-46f5-bcfa-9357925458e4" (UID: "ef1f600f-0137-46f5-bcfa-9357925458e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:16:31.376000 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:31.372454    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef1f600f-0137-46f5-bcfa-9357925458e4-maas-api-tls" pod "ef1f600f-0137-46f5-bcfa-9357925458e4" (UID: "ef1f600f-0137-46f5-bcfa-9357925458e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:16:31.383890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:31.383855    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef1f600f-0137-46f5-bcfa-9357925458e4-metrics-certs" pod "ef1f600f-0137-46f5-bcfa-9357925458e4" (UID: "ef1f600f-0137-46f5-bcfa-9357925458e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:16:31.384887 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:31.384855    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1f600f-0137-46f5-bcfa-9357925458e4-kube-api-access-mfhnd" pod "ef1f600f-0137-46f5-bcfa-9357925458e4" (UID: "ef1f600f-0137-46f5-bcfa-9357925458e4"). InnerVolumeSpecName "kube-api-access-mfhnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:16:31.455625 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:31.455594    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfhnd\" (UniqueName: \"kubernetes.io/projected/ef1f600f-0137-46f5-bcfa-9357925458e4-kube-api-access-mfhnd\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:16:31.455625 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:31.455624    5875 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef1f600f-0137-46f5-bcfa-9357925458e4-openshift-service-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:16:31.455827 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:31.455635    5875 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef1f600f-0137-46f5-bcfa-9357925458e4-metrics-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:16:31.455827 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:31.455644    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef1f600f-0137-46f5-bcfa-9357925458e4-maas-api-tls\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:16:31.838114 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:31.838077    5875 generic.go:356] "Generic (PLEG): container finished" podID="ef1f600f-0137-46f5-bcfa-9357925458e4" containerID="75e5a5f82f7810e6b3b0f1ac778c4e7c47ae38968dcbd7acbfade1502deeb720" exitCode=0
Nov 02 02:16:31.838325 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:31.838128    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd"
Nov 02 02:16:31.838325 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:31.838170    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd" event={"ID":"ef1f600f-0137-46f5-bcfa-9357925458e4","Type":"ContainerDied","Data":"75e5a5f82f7810e6b3b0f1ac778c4e7c47ae38968dcbd7acbfade1502deeb720"}
Nov 02 02:16:31.838325 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:31.838249    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd" event={"ID":"ef1f600f-0137-46f5-bcfa-9357925458e4","Type":"ContainerDied","Data":"026dd022d4bd3b71c4d852f84d21993c162c0a80f9e2544b6d976868a2b1b10b"}
Nov 02 02:16:31.838325 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:31.838272    5875 scope.go:122] "RemoveContainer" containerID="75e5a5f82f7810e6b3b0f1ac778c4e7c47ae38968dcbd7acbfade1502deeb720"
Nov 02 02:16:31.849050 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:31.849032    5875 scope.go:122] "RemoveContainer" containerID="75e5a5f82f7810e6b3b0f1ac778c4e7c47ae38968dcbd7acbfade1502deeb720"
Nov 02 02:16:31.849401 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:16:31.849380    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75e5a5f82f7810e6b3b0f1ac778c4e7c47ae38968dcbd7acbfade1502deeb720\": container with ID starting with 75e5a5f82f7810e6b3b0f1ac778c4e7c47ae38968dcbd7acbfade1502deeb720 not found: ID does not exist" containerID="75e5a5f82f7810e6b3b0f1ac778c4e7c47ae38968dcbd7acbfade1502deeb720"
Nov 02 02:16:31.849465 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:31.849412    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75e5a5f82f7810e6b3b0f1ac778c4e7c47ae38968dcbd7acbfade1502deeb720"} err="failed to get container status \"75e5a5f82f7810e6b3b0f1ac778c4e7c47ae38968dcbd7acbfade1502deeb720\": rpc error: code = NotFound desc = could not find container \"75e5a5f82f7810e6b3b0f1ac778c4e7c47ae38968dcbd7acbfade1502deeb720\": container with ID starting with 75e5a5f82f7810e6b3b0f1ac778c4e7c47ae38968dcbd7acbfade1502deeb720 not found: ID does not exist"
Nov 02 02:16:32.001116 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:32.001079    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd"]
Nov 02 02:16:32.003148 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:32.003127    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c942d-5c4f84778c-trlgd"]
Nov 02 02:16:32.693773 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:32.693740    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef1f600f-0137-46f5-bcfa-9357925458e4" path="/var/lib/kubelet/pods/ef1f600f-0137-46f5-bcfa-9357925458e4/volumes"
Nov 02 02:16:33.084949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:33.084885    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg"]
Nov 02 02:16:33.134312 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:33.134278    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c942d-l5vsg"]
Nov 02 02:16:34.693038 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:34.693001    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17668486-eec6-4551-a99c-7efd65458f0a" path="/var/lib/kubelet/pods/17668486-eec6-4551-a99c-7efd65458f0a/volumes"
Nov 02 02:16:37.181516 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:37.181473    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:16:40.586856 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.586820    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb"]
Nov 02 02:16:40.587404 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.587359    5875 cpu_manager.go:414] "containerMap: removing container" podUID="ef1f600f-0137-46f5-bcfa-9357925458e4" containerName="maas-api"
Nov 02 02:16:40.587404 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.587371    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef1f600f-0137-46f5-bcfa-9357925458e4" containerName="maas-api"
Nov 02 02:16:40.587404 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.587381    5875 cpu_manager.go:414] "containerMap: removing container" podUID="17668486-eec6-4551-a99c-7efd65458f0a" containerName="revoke-keys"
Nov 02 02:16:40.587404 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.587386    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17668486-eec6-4551-a99c-7efd65458f0a" containerName="revoke-keys"
Nov 02 02:16:40.587404 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.587404    5875 cpu_manager.go:414] "containerMap: removing container" podUID="c30b1225-cfec-4f91-9fa0-ab84afcb0598" containerName="main"
Nov 02 02:16:40.587607 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.587409    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c30b1225-cfec-4f91-9fa0-ab84afcb0598" containerName="main"
Nov 02 02:16:40.587607 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.587497    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef1f600f-0137-46f5-bcfa-9357925458e4" containerName="maas-api"
Nov 02 02:16:40.587607 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.587506    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="17668486-eec6-4551-a99c-7efd65458f0a" containerName="revoke-keys"
Nov 02 02:16:40.587607 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.587515    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="c30b1225-cfec-4f91-9fa0-ab84afcb0598" containerName="main"
Nov 02 02:16:40.591935 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.591918    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb"
Nov 02 02:16:40.644904 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.644872    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:16:40.645130 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.645028    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:16:40.659413 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.659378    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d56d1cfa-329f-4259-b6e5-5ddafb07f848-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb\" (UID: \"d56d1cfa-329f-4259-b6e5-5ddafb07f848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb"
Nov 02 02:16:40.659613 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.659450    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d56d1cfa-329f-4259-b6e5-5ddafb07f848-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb\" (UID: \"d56d1cfa-329f-4259-b6e5-5ddafb07f848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb"
Nov 02 02:16:40.659613 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.659520    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj5lb\" (UniqueName: \"kubernetes.io/projected/d56d1cfa-329f-4259-b6e5-5ddafb07f848-kube-api-access-bj5lb\") pod \"maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb\" (UID: \"d56d1cfa-329f-4259-b6e5-5ddafb07f848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb"
Nov 02 02:16:40.694365 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.694337    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:16:40.694559 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.694411    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:16:40.696689 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.696671    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb"]
Nov 02 02:16:40.761014 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.760969    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d56d1cfa-329f-4259-b6e5-5ddafb07f848-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb\" (UID: \"d56d1cfa-329f-4259-b6e5-5ddafb07f848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb"
Nov 02 02:16:40.761014 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.761017    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5lb\" (UniqueName: \"kubernetes.io/projected/d56d1cfa-329f-4259-b6e5-5ddafb07f848-kube-api-access-bj5lb\") pod \"maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb\" (UID: \"d56d1cfa-329f-4259-b6e5-5ddafb07f848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb"
Nov 02 02:16:40.761255 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.761116    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d56d1cfa-329f-4259-b6e5-5ddafb07f848-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb\" (UID: \"d56d1cfa-329f-4259-b6e5-5ddafb07f848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb"
Nov 02 02:16:40.761658 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.761627    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d56d1cfa-329f-4259-b6e5-5ddafb07f848-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb\" (UID: \"d56d1cfa-329f-4259-b6e5-5ddafb07f848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb"
Nov 02 02:16:40.763538 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.763513    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d56d1cfa-329f-4259-b6e5-5ddafb07f848-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb\" (UID: \"d56d1cfa-329f-4259-b6e5-5ddafb07f848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb"
Nov 02 02:16:40.821718 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.821685    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj5lb\" (UniqueName: \"kubernetes.io/projected/d56d1cfa-329f-4259-b6e5-5ddafb07f848-kube-api-access-bj5lb\") pod \"maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb\" (UID: \"d56d1cfa-329f-4259-b6e5-5ddafb07f848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb"
Nov 02 02:16:40.905631 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:40.905559    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb"
Nov 02 02:16:41.279915 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:41.279890    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb"]
Nov 02 02:16:41.282471 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:16:41.282441    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56d1cfa_329f_4259_b6e5_5ddafb07f848.slice/crio-5a7e6eaa76f788e9f4b4ce1a3c26f7233143ed9b3a889387fa565ebcdb1d5828 WatchSource:0}: Error finding container 5a7e6eaa76f788e9f4b4ce1a3c26f7233143ed9b3a889387fa565ebcdb1d5828: Status 404 returned error can't find the container with id 5a7e6eaa76f788e9f4b4ce1a3c26f7233143ed9b3a889387fa565ebcdb1d5828
Nov 02 02:16:41.318626 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:16:41.318592    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:16:41.903546 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:41.903455    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb" event={"ID":"d56d1cfa-329f-4259-b6e5-5ddafb07f848","Type":"ContainerStarted","Data":"ab50c6af463e0296d3bef9735b3de9254a9994ab16c8fd8eeb05c6fa51296ab8"}
Nov 02 02:16:41.903546 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:41.903509    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb" event={"ID":"d56d1cfa-329f-4259-b6e5-5ddafb07f848","Type":"ContainerStarted","Data":"5a7e6eaa76f788e9f4b4ce1a3c26f7233143ed9b3a889387fa565ebcdb1d5828"}
Nov 02 02:16:42.020120 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:42.020055    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb" podStartSLOduration=2.02003963 podStartE2EDuration="2.02003963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:16:42.018347894 +0000 UTC m=+2649.459384390" watchObservedRunningTime="2026-11-02 02:16:42.02003963 +0000 UTC m=+2649.461076127"
Nov 02 02:16:42.912044 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:42.912004    5875 generic.go:356] "Generic (PLEG): container finished" podID="d56d1cfa-329f-4259-b6e5-5ddafb07f848" containerID="ab50c6af463e0296d3bef9735b3de9254a9994ab16c8fd8eeb05c6fa51296ab8" exitCode=0
Nov 02 02:16:42.912494 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:42.912088    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb" event={"ID":"d56d1cfa-329f-4259-b6e5-5ddafb07f848","Type":"ContainerDied","Data":"ab50c6af463e0296d3bef9735b3de9254a9994ab16c8fd8eeb05c6fa51296ab8"}
Nov 02 02:16:44.068717 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:44.068690    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb"
Nov 02 02:16:44.199207 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:44.199125    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d56d1cfa-329f-4259-b6e5-5ddafb07f848-kube-api-access-bj5lb\" (UniqueName: \"kubernetes.io/projected/d56d1cfa-329f-4259-b6e5-5ddafb07f848-kube-api-access-bj5lb\") pod \"d56d1cfa-329f-4259-b6e5-5ddafb07f848\" (UID: \"d56d1cfa-329f-4259-b6e5-5ddafb07f848\") "
Nov 02 02:16:44.199207 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:44.199202    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d56d1cfa-329f-4259-b6e5-5ddafb07f848-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d56d1cfa-329f-4259-b6e5-5ddafb07f848-maas-api-xks-ca\") pod \"d56d1cfa-329f-4259-b6e5-5ddafb07f848\" (UID: \"d56d1cfa-329f-4259-b6e5-5ddafb07f848\") "
Nov 02 02:16:44.199477 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:44.199292    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d56d1cfa-329f-4259-b6e5-5ddafb07f848-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d56d1cfa-329f-4259-b6e5-5ddafb07f848-maas-api-openshift-ca\") pod \"d56d1cfa-329f-4259-b6e5-5ddafb07f848\" (UID: \"d56d1cfa-329f-4259-b6e5-5ddafb07f848\") "
Nov 02 02:16:44.199719 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:44.199688    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d56d1cfa-329f-4259-b6e5-5ddafb07f848-maas-api-openshift-ca" pod "d56d1cfa-329f-4259-b6e5-5ddafb07f848" (UID: "d56d1cfa-329f-4259-b6e5-5ddafb07f848"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:16:44.201410 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:44.201383    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d56d1cfa-329f-4259-b6e5-5ddafb07f848-kube-api-access-bj5lb" pod "d56d1cfa-329f-4259-b6e5-5ddafb07f848" (UID: "d56d1cfa-329f-4259-b6e5-5ddafb07f848"). InnerVolumeSpecName "kube-api-access-bj5lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:16:44.201531 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:44.201465    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d56d1cfa-329f-4259-b6e5-5ddafb07f848-maas-api-xks-ca" pod "d56d1cfa-329f-4259-b6e5-5ddafb07f848" (UID: "d56d1cfa-329f-4259-b6e5-5ddafb07f848"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:16:44.301034 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:44.301003    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj5lb\" (UniqueName: \"kubernetes.io/projected/d56d1cfa-329f-4259-b6e5-5ddafb07f848-kube-api-access-bj5lb\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:16:44.301034 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:44.301033    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d56d1cfa-329f-4259-b6e5-5ddafb07f848-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:16:44.301272 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:44.301047    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d56d1cfa-329f-4259-b6e5-5ddafb07f848-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:16:44.925000 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:44.924964    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb"
Nov 02 02:16:44.925000 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:44.924990    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb" event={"ID":"d56d1cfa-329f-4259-b6e5-5ddafb07f848","Type":"ContainerDied","Data":"5a7e6eaa76f788e9f4b4ce1a3c26f7233143ed9b3a889387fa565ebcdb1d5828"}
Nov 02 02:16:44.925245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:44.925042    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a7e6eaa76f788e9f4b4ce1a3c26f7233143ed9b3a889387fa565ebcdb1d5828"
Nov 02 02:16:52.055361 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:52.055326    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm"]
Nov 02 02:16:52.160186 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:52.160152    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls"]
Nov 02 02:16:57.056565 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:57.056525    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm" podUID="0fa3f0ef-b33a-4fda-9e31-5393367463e2" containerName="payload-processing" containerID="cri-o://0a8b7a7a2adfb721a57a171e3f36dff77027dc44d026150974e57e36da92d08b" gracePeriod=25
Nov 02 02:16:57.160965 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:57.160926    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls" podUID="ad5288b9-b2da-41be-830a-72c877a15faf" containerName="payload-pre-processing" containerID="cri-o://0f2d5a6bbcaf74c9380464df41b94313328f40d990b29bc04310c5a73cbf58d3" gracePeriod=25
Nov 02 02:16:58.338360 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:58.338322    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls" podUID="ad5288b9-b2da-41be-830a-72c877a15faf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:16:58.338790 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:16:58.338322    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm" podUID="0fa3f0ef-b33a-4fda-9e31-5393367463e2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:17:03.778190 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:03.778154    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb"]
Nov 02 02:17:03.827850 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:03.827820    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4211c0-wnnkb"]
Nov 02 02:17:04.693784 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:04.693751    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d56d1cfa-329f-4259-b6e5-5ddafb07f848" path="/var/lib/kubelet/pods/d56d1cfa-329f-4259-b6e5-5ddafb07f848/volumes"
Nov 02 02:17:07.303769 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:07.303735    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:17:08.337691 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:08.337655    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls" podUID="ad5288b9-b2da-41be-830a-72c877a15faf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:17:08.337691 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:08.337672    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm" podUID="0fa3f0ef-b33a-4fda-9e31-5393367463e2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:17:10.711290 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:10.711261    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"]
Nov 02 02:17:10.711738 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:10.711462    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn" podUID="93d9b083-f014-4998-91b1-a90d3cce473c" containerName="istio-proxy" containerID="cri-o://f6cbbed20ef8629aae1323276a23d81d4490b6dd0de6e5442e00795af56c8bd2" gracePeriod=30
Nov 02 02:17:11.319663 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:17:11.319620    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:17:11.697558 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:11.697469    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9"]
Nov 02 02:17:11.697975 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:11.697961    5875 cpu_manager.go:414] "containerMap: removing container" podUID="d56d1cfa-329f-4259-b6e5-5ddafb07f848" containerName="revoke-keys"
Nov 02 02:17:11.697975 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:11.697975    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d56d1cfa-329f-4259-b6e5-5ddafb07f848" containerName="revoke-keys"
Nov 02 02:17:11.698109 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:11.698076    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="d56d1cfa-329f-4259-b6e5-5ddafb07f848" containerName="revoke-keys"
Nov 02 02:17:11.701511 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:11.701488    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9"
Nov 02 02:17:11.756147 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:11.756102    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:17:11.805736 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:11.805708    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:17:11.805736 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:11.805704    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:17:11.805736 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:11.805723    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:17:11.808096 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:11.808075    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9"]
Nov 02 02:17:11.815410 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:11.815385    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e34877d3-dc45-4fd4-8d11-bb85c91458af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9\" (UID: \"e34877d3-dc45-4fd4-8d11-bb85c91458af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9"
Nov 02 02:17:11.815493 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:11.815414    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e34877d3-dc45-4fd4-8d11-bb85c91458af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9\" (UID: \"e34877d3-dc45-4fd4-8d11-bb85c91458af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9"
Nov 02 02:17:11.815493 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:11.815446    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh7dq\" (UniqueName: \"kubernetes.io/projected/e34877d3-dc45-4fd4-8d11-bb85c91458af-kube-api-access-gh7dq\") pod \"maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9\" (UID: \"e34877d3-dc45-4fd4-8d11-bb85c91458af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9"
Nov 02 02:17:11.917178 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:11.917143    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e34877d3-dc45-4fd4-8d11-bb85c91458af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9\" (UID: \"e34877d3-dc45-4fd4-8d11-bb85c91458af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9"
Nov 02 02:17:11.917178 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:11.917173    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e34877d3-dc45-4fd4-8d11-bb85c91458af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9\" (UID: \"e34877d3-dc45-4fd4-8d11-bb85c91458af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9"
Nov 02 02:17:11.917384 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:11.917200    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh7dq\" (UniqueName: \"kubernetes.io/projected/e34877d3-dc45-4fd4-8d11-bb85c91458af-kube-api-access-gh7dq\") pod \"maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9\" (UID: \"e34877d3-dc45-4fd4-8d11-bb85c91458af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9"
Nov 02 02:17:11.917967 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:11.917906    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e34877d3-dc45-4fd4-8d11-bb85c91458af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9\" (UID: \"e34877d3-dc45-4fd4-8d11-bb85c91458af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9"
Nov 02 02:17:11.919793 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:11.919769    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e34877d3-dc45-4fd4-8d11-bb85c91458af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9\" (UID: \"e34877d3-dc45-4fd4-8d11-bb85c91458af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9"
Nov 02 02:17:11.977790 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:11.977758    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh7dq\" (UniqueName: \"kubernetes.io/projected/e34877d3-dc45-4fd4-8d11-bb85c91458af-kube-api-access-gh7dq\") pod \"maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9\" (UID: \"e34877d3-dc45-4fd4-8d11-bb85c91458af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9"
Nov 02 02:17:12.017768 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:12.017746    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9"
Nov 02 02:17:12.394148 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:12.394121    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9"]
Nov 02 02:17:12.396119 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:17:12.396089    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode34877d3_dc45_4fd4_8d11_bb85c91458af.slice/crio-4e90aa7cee4a884e4dfa92195e1260f106ea9ae39fb2317495520becd29575d4 WatchSource:0}: Error finding container 4e90aa7cee4a884e4dfa92195e1260f106ea9ae39fb2317495520becd29575d4: Status 404 returned error can't find the container with id 4e90aa7cee4a884e4dfa92195e1260f106ea9ae39fb2317495520becd29575d4
Nov 02 02:17:13.102605 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:13.102504    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9" event={"ID":"e34877d3-dc45-4fd4-8d11-bb85c91458af","Type":"ContainerStarted","Data":"4f421defb1f349a3772a5f2325269f30019583d1d2e2357f47fa27be51d1bb56"}
Nov 02 02:17:13.102605 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:13.102572    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9" event={"ID":"e34877d3-dc45-4fd4-8d11-bb85c91458af","Type":"ContainerStarted","Data":"4e90aa7cee4a884e4dfa92195e1260f106ea9ae39fb2317495520becd29575d4"}
Nov 02 02:17:13.220912 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:13.220858    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9" podStartSLOduration=2.220841692 podStartE2EDuration="2.220841692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:17:13.219533437 +0000 UTC m=+2680.660569948" watchObservedRunningTime="2026-11-02 02:17:13.220841692 +0000 UTC m=+2680.661878193"
Nov 02 02:17:14.110837 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:14.110797    5875 generic.go:356] "Generic (PLEG): container finished" podID="e34877d3-dc45-4fd4-8d11-bb85c91458af" containerID="4f421defb1f349a3772a5f2325269f30019583d1d2e2357f47fa27be51d1bb56" exitCode=0
Nov 02 02:17:14.111277 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:14.110893    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9" event={"ID":"e34877d3-dc45-4fd4-8d11-bb85c91458af","Type":"ContainerDied","Data":"4f421defb1f349a3772a5f2325269f30019583d1d2e2357f47fa27be51d1bb56"}
Nov 02 02:17:15.258599 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:15.258578    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9"
Nov 02 02:17:15.351136 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:15.351112    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e34877d3-dc45-4fd4-8d11-bb85c91458af-kube-api-access-gh7dq\" (UniqueName: \"kubernetes.io/projected/e34877d3-dc45-4fd4-8d11-bb85c91458af-kube-api-access-gh7dq\") pod \"e34877d3-dc45-4fd4-8d11-bb85c91458af\" (UID: \"e34877d3-dc45-4fd4-8d11-bb85c91458af\") "
Nov 02 02:17:15.351318 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:15.351298    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e34877d3-dc45-4fd4-8d11-bb85c91458af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e34877d3-dc45-4fd4-8d11-bb85c91458af-maas-api-xks-ca\") pod \"e34877d3-dc45-4fd4-8d11-bb85c91458af\" (UID: \"e34877d3-dc45-4fd4-8d11-bb85c91458af\") "
Nov 02 02:17:15.351373 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:15.351340    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e34877d3-dc45-4fd4-8d11-bb85c91458af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e34877d3-dc45-4fd4-8d11-bb85c91458af-maas-api-openshift-ca\") pod \"e34877d3-dc45-4fd4-8d11-bb85c91458af\" (UID: \"e34877d3-dc45-4fd4-8d11-bb85c91458af\") "
Nov 02 02:17:15.351714 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:15.351689    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e34877d3-dc45-4fd4-8d11-bb85c91458af-maas-api-openshift-ca" pod "e34877d3-dc45-4fd4-8d11-bb85c91458af" (UID: "e34877d3-dc45-4fd4-8d11-bb85c91458af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:17:15.353310 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:15.353288    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e34877d3-dc45-4fd4-8d11-bb85c91458af-kube-api-access-gh7dq" pod "e34877d3-dc45-4fd4-8d11-bb85c91458af" (UID: "e34877d3-dc45-4fd4-8d11-bb85c91458af"). InnerVolumeSpecName "kube-api-access-gh7dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:17:15.353417 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:15.353394    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e34877d3-dc45-4fd4-8d11-bb85c91458af-maas-api-xks-ca" pod "e34877d3-dc45-4fd4-8d11-bb85c91458af" (UID: "e34877d3-dc45-4fd4-8d11-bb85c91458af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:17:15.452909 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:15.452824    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e34877d3-dc45-4fd4-8d11-bb85c91458af-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:15.452909 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:15.452858    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e34877d3-dc45-4fd4-8d11-bb85c91458af-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:15.452909 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:15.452868    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh7dq\" (UniqueName: \"kubernetes.io/projected/e34877d3-dc45-4fd4-8d11-bb85c91458af-kube-api-access-gh7dq\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:15.960586 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:15.960560    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:17:16.059036 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.058953    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93d9b083-f014-4998-91b1-a90d3cce473c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93d9b083-f014-4998-91b1-a90d3cce473c-istiod-ca-cert\") pod \"93d9b083-f014-4998-91b1-a90d3cce473c\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") "
Nov 02 02:17:16.059036 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.059016    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-istio-envoy\") pod \"93d9b083-f014-4998-91b1-a90d3cce473c\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") "
Nov 02 02:17:16.059267 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.059151    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93d9b083-f014-4998-91b1-a90d3cce473c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93d9b083-f014-4998-91b1-a90d3cce473c-istio-podinfo\") pod \"93d9b083-f014-4998-91b1-a90d3cce473c\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") "
Nov 02 02:17:16.059267 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.059247    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-workload-certs\") pod \"93d9b083-f014-4998-91b1-a90d3cce473c\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") "
Nov 02 02:17:16.059389 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.059282    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93d9b083-f014-4998-91b1-a90d3cce473c-kube-api-access-zh42p\" (UniqueName: \"kubernetes.io/projected/93d9b083-f014-4998-91b1-a90d3cce473c-kube-api-access-zh42p\") pod \"93d9b083-f014-4998-91b1-a90d3cce473c\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") "
Nov 02 02:17:16.059389 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.059344    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-istio-data\") pod \"93d9b083-f014-4998-91b1-a90d3cce473c\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") "
Nov 02 02:17:16.059508 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.059401    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-credential-socket\") pod \"93d9b083-f014-4998-91b1-a90d3cce473c\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") "
Nov 02 02:17:16.059508 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.059424    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93d9b083-f014-4998-91b1-a90d3cce473c-istiod-ca-cert" pod "93d9b083-f014-4998-91b1-a90d3cce473c" (UID: "93d9b083-f014-4998-91b1-a90d3cce473c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:17:16.059508 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.059435    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-workload-socket\") pod \"93d9b083-f014-4998-91b1-a90d3cce473c\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") "
Nov 02 02:17:16.059508 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.059477    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93d9b083-f014-4998-91b1-a90d3cce473c-istio-token\" (UniqueName: \"kubernetes.io/projected/93d9b083-f014-4998-91b1-a90d3cce473c-istio-token\") pod \"93d9b083-f014-4998-91b1-a90d3cce473c\" (UID: \"93d9b083-f014-4998-91b1-a90d3cce473c\") "
Nov 02 02:17:16.059730 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.059693    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-credential-socket" pod "93d9b083-f014-4998-91b1-a90d3cce473c" (UID: "93d9b083-f014-4998-91b1-a90d3cce473c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:17:16.059730 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.059706    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-workload-socket" pod "93d9b083-f014-4998-91b1-a90d3cce473c" (UID: "93d9b083-f014-4998-91b1-a90d3cce473c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:17:16.059966 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.059938    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-workload-certs" pod "93d9b083-f014-4998-91b1-a90d3cce473c" (UID: "93d9b083-f014-4998-91b1-a90d3cce473c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:17:16.060198 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.060037    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-istio-data" pod "93d9b083-f014-4998-91b1-a90d3cce473c" (UID: "93d9b083-f014-4998-91b1-a90d3cce473c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:17:16.060198 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.060080    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93d9b083-f014-4998-91b1-a90d3cce473c-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:16.060198 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.060093    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:16.060198 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.060106    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:16.060198 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.060123    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:16.061656 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.061629    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-istio-envoy" pod "93d9b083-f014-4998-91b1-a90d3cce473c" (UID: "93d9b083-f014-4998-91b1-a90d3cce473c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:17:16.061894 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.061873    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93d9b083-f014-4998-91b1-a90d3cce473c-istio-podinfo" pod "93d9b083-f014-4998-91b1-a90d3cce473c" (UID: "93d9b083-f014-4998-91b1-a90d3cce473c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:17:16.061970 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.061954    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d9b083-f014-4998-91b1-a90d3cce473c-istio-token" pod "93d9b083-f014-4998-91b1-a90d3cce473c" (UID: "93d9b083-f014-4998-91b1-a90d3cce473c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:17:16.062455 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.062432    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d9b083-f014-4998-91b1-a90d3cce473c-kube-api-access-zh42p" pod "93d9b083-f014-4998-91b1-a90d3cce473c" (UID: "93d9b083-f014-4998-91b1-a90d3cce473c"). InnerVolumeSpecName "kube-api-access-zh42p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:17:16.133809 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.133778    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9"
Nov 02 02:17:16.133809 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.133799    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9" event={"ID":"e34877d3-dc45-4fd4-8d11-bb85c91458af","Type":"ContainerDied","Data":"4e90aa7cee4a884e4dfa92195e1260f106ea9ae39fb2317495520becd29575d4"}
Nov 02 02:17:16.134104 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.133884    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e90aa7cee4a884e4dfa92195e1260f106ea9ae39fb2317495520becd29575d4"
Nov 02 02:17:16.136910 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.136884    5875 generic.go:356] "Generic (PLEG): container finished" podID="93d9b083-f014-4998-91b1-a90d3cce473c" containerID="f6cbbed20ef8629aae1323276a23d81d4490b6dd0de6e5442e00795af56c8bd2" exitCode=0
Nov 02 02:17:16.137059 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.136956    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"
Nov 02 02:17:16.137059 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.137045    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn" event={"ID":"93d9b083-f014-4998-91b1-a90d3cce473c","Type":"ContainerDied","Data":"f6cbbed20ef8629aae1323276a23d81d4490b6dd0de6e5442e00795af56c8bd2"}
Nov 02 02:17:16.137194 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.137084    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn" event={"ID":"93d9b083-f014-4998-91b1-a90d3cce473c","Type":"ContainerDied","Data":"4dc49dc6f5906862e676b5b65356113ad6f1f8f23a9906ea4df1f9089001e48e"}
Nov 02 02:17:16.137194 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.137112    5875 scope.go:122] "RemoveContainer" containerID="f6cbbed20ef8629aae1323276a23d81d4490b6dd0de6e5442e00795af56c8bd2"
Nov 02 02:17:16.148892 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.148874    5875 scope.go:122] "RemoveContainer" containerID="28ed2b891f119c8f2542e0c5598765f9ec0988beec097bc0617f6ffceb5c4f5c"
Nov 02 02:17:16.157886 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.157864    5875 scope.go:122] "RemoveContainer" containerID="f6cbbed20ef8629aae1323276a23d81d4490b6dd0de6e5442e00795af56c8bd2"
Nov 02 02:17:16.158173 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:17:16.158153    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6cbbed20ef8629aae1323276a23d81d4490b6dd0de6e5442e00795af56c8bd2\": container with ID starting with f6cbbed20ef8629aae1323276a23d81d4490b6dd0de6e5442e00795af56c8bd2 not found: ID does not exist" containerID="f6cbbed20ef8629aae1323276a23d81d4490b6dd0de6e5442e00795af56c8bd2"
Nov 02 02:17:16.158263 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.158182    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6cbbed20ef8629aae1323276a23d81d4490b6dd0de6e5442e00795af56c8bd2"} err="failed to get container status \"f6cbbed20ef8629aae1323276a23d81d4490b6dd0de6e5442e00795af56c8bd2\": rpc error: code = NotFound desc = could not find container \"f6cbbed20ef8629aae1323276a23d81d4490b6dd0de6e5442e00795af56c8bd2\": container with ID starting with f6cbbed20ef8629aae1323276a23d81d4490b6dd0de6e5442e00795af56c8bd2 not found: ID does not exist"
Nov 02 02:17:16.158263 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.158201    5875 scope.go:122] "RemoveContainer" containerID="28ed2b891f119c8f2542e0c5598765f9ec0988beec097bc0617f6ffceb5c4f5c"
Nov 02 02:17:16.158488 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:17:16.158464    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28ed2b891f119c8f2542e0c5598765f9ec0988beec097bc0617f6ffceb5c4f5c\": container with ID starting with 28ed2b891f119c8f2542e0c5598765f9ec0988beec097bc0617f6ffceb5c4f5c not found: ID does not exist" containerID="28ed2b891f119c8f2542e0c5598765f9ec0988beec097bc0617f6ffceb5c4f5c"
Nov 02 02:17:16.158604 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.158491    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28ed2b891f119c8f2542e0c5598765f9ec0988beec097bc0617f6ffceb5c4f5c"} err="failed to get container status \"28ed2b891f119c8f2542e0c5598765f9ec0988beec097bc0617f6ffceb5c4f5c\": rpc error: code = NotFound desc = could not find container \"28ed2b891f119c8f2542e0c5598765f9ec0988beec097bc0617f6ffceb5c4f5c\": container with ID starting with 28ed2b891f119c8f2542e0c5598765f9ec0988beec097bc0617f6ffceb5c4f5c not found: ID does not exist"
Nov 02 02:17:16.161052 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.161032    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93d9b083-f014-4998-91b1-a90d3cce473c-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:16.161147 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.161056    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:16.161147 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.161065    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93d9b083-f014-4998-91b1-a90d3cce473c-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:16.161147 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.161075    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh42p\" (UniqueName: \"kubernetes.io/projected/93d9b083-f014-4998-91b1-a90d3cce473c-kube-api-access-zh42p\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:16.161147 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.161084    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93d9b083-f014-4998-91b1-a90d3cce473c-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:16.355108 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.355020    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"]
Nov 02 02:17:16.358749 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.358726    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4211c0-openshift-default-85966f878c-blgmn"]
Nov 02 02:17:16.694048 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:16.693973    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93d9b083-f014-4998-91b1-a90d3cce473c" path="/var/lib/kubelet/pods/93d9b083-f014-4998-91b1-a90d3cce473c/volumes"
Nov 02 02:17:17.324737 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:17.324663    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm"
Nov 02 02:17:17.430182 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:17.430160    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls"
Nov 02 02:17:17.473543 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:17.473512    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fa3f0ef-b33a-4fda-9e31-5393367463e2-kube-api-access-cggdj\" (UniqueName: \"kubernetes.io/projected/0fa3f0ef-b33a-4fda-9e31-5393367463e2-kube-api-access-cggdj\") pod \"0fa3f0ef-b33a-4fda-9e31-5393367463e2\" (UID: \"0fa3f0ef-b33a-4fda-9e31-5393367463e2\") "
Nov 02 02:17:17.473742 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:17.473656    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fa3f0ef-b33a-4fda-9e31-5393367463e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa3f0ef-b33a-4fda-9e31-5393367463e2-plugins-config-volume\") pod \"0fa3f0ef-b33a-4fda-9e31-5393367463e2\" (UID: \"0fa3f0ef-b33a-4fda-9e31-5393367463e2\") "
Nov 02 02:17:17.474090 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:17.474047    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa3f0ef-b33a-4fda-9e31-5393367463e2-plugins-config-volume" pod "0fa3f0ef-b33a-4fda-9e31-5393367463e2" (UID: "0fa3f0ef-b33a-4fda-9e31-5393367463e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:17:17.475837 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:17.475814    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa3f0ef-b33a-4fda-9e31-5393367463e2-kube-api-access-cggdj" pod "0fa3f0ef-b33a-4fda-9e31-5393367463e2" (UID: "0fa3f0ef-b33a-4fda-9e31-5393367463e2"). InnerVolumeSpecName "kube-api-access-cggdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:17:17.575238 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:17.575140    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad5288b9-b2da-41be-830a-72c877a15faf-kube-api-access-jqmq2\" (UniqueName: \"kubernetes.io/projected/ad5288b9-b2da-41be-830a-72c877a15faf-kube-api-access-jqmq2\") pod \"ad5288b9-b2da-41be-830a-72c877a15faf\" (UID: \"ad5288b9-b2da-41be-830a-72c877a15faf\") "
Nov 02 02:17:17.575388 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:17.575293    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad5288b9-b2da-41be-830a-72c877a15faf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad5288b9-b2da-41be-830a-72c877a15faf-plugins-config-volume\") pod \"ad5288b9-b2da-41be-830a-72c877a15faf\" (UID: \"ad5288b9-b2da-41be-830a-72c877a15faf\") "
Nov 02 02:17:17.575634 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:17.575612    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad5288b9-b2da-41be-830a-72c877a15faf-plugins-config-volume" pod "ad5288b9-b2da-41be-830a-72c877a15faf" (UID: "ad5288b9-b2da-41be-830a-72c877a15faf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:17:17.575752 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:17.575737    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad5288b9-b2da-41be-830a-72c877a15faf-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:17.575827 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:17.575754    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cggdj\" (UniqueName: \"kubernetes.io/projected/0fa3f0ef-b33a-4fda-9e31-5393367463e2-kube-api-access-cggdj\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:17.575827 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:17.575763    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa3f0ef-b33a-4fda-9e31-5393367463e2-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:17.577711 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:17.577685    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5288b9-b2da-41be-830a-72c877a15faf-kube-api-access-jqmq2" pod "ad5288b9-b2da-41be-830a-72c877a15faf" (UID: "ad5288b9-b2da-41be-830a-72c877a15faf"). InnerVolumeSpecName "kube-api-access-jqmq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:17:17.676615 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:17.676578    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqmq2\" (UniqueName: \"kubernetes.io/projected/ad5288b9-b2da-41be-830a-72c877a15faf-kube-api-access-jqmq2\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:18.150779 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:18.150738    5875 generic.go:356] "Generic (PLEG): container finished" podID="ad5288b9-b2da-41be-830a-72c877a15faf" containerID="0f2d5a6bbcaf74c9380464df41b94313328f40d990b29bc04310c5a73cbf58d3" exitCode=0
Nov 02 02:17:18.150941 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:18.150840    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls"
Nov 02 02:17:18.150941 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:18.150880    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls" event={"ID":"ad5288b9-b2da-41be-830a-72c877a15faf","Type":"ContainerDied","Data":"0f2d5a6bbcaf74c9380464df41b94313328f40d990b29bc04310c5a73cbf58d3"}
Nov 02 02:17:18.151066 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:18.150942    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls" event={"ID":"ad5288b9-b2da-41be-830a-72c877a15faf","Type":"ContainerDied","Data":"5433587b5a8f79178b8fb750da8b81bdd38417908b9644fa2d844742ae623f20"}
Nov 02 02:17:18.151066 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:18.150974    5875 scope.go:122] "RemoveContainer" containerID="0f2d5a6bbcaf74c9380464df41b94313328f40d990b29bc04310c5a73cbf58d3"
Nov 02 02:17:18.155599 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:18.152321    5875 generic.go:356] "Generic (PLEG): container finished" podID="0fa3f0ef-b33a-4fda-9e31-5393367463e2" containerID="0a8b7a7a2adfb721a57a171e3f36dff77027dc44d026150974e57e36da92d08b" exitCode=0
Nov 02 02:17:18.155599 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:18.152391    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm"
Nov 02 02:17:18.155599 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:18.152429    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm" event={"ID":"0fa3f0ef-b33a-4fda-9e31-5393367463e2","Type":"ContainerDied","Data":"0a8b7a7a2adfb721a57a171e3f36dff77027dc44d026150974e57e36da92d08b"}
Nov 02 02:17:18.155599 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:18.152471    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm" event={"ID":"0fa3f0ef-b33a-4fda-9e31-5393367463e2","Type":"ContainerDied","Data":"ee222f14d0d517b43879cef5a381338f2456cb80d75762138e1dfe9a7436838a"}
Nov 02 02:17:18.160603 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:18.160473    5875 scope.go:122] "RemoveContainer" containerID="0f2d5a6bbcaf74c9380464df41b94313328f40d990b29bc04310c5a73cbf58d3"
Nov 02 02:17:18.160756 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:17:18.160730    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f2d5a6bbcaf74c9380464df41b94313328f40d990b29bc04310c5a73cbf58d3\": container with ID starting with 0f2d5a6bbcaf74c9380464df41b94313328f40d990b29bc04310c5a73cbf58d3 not found: ID does not exist" containerID="0f2d5a6bbcaf74c9380464df41b94313328f40d990b29bc04310c5a73cbf58d3"
Nov 02 02:17:18.160884 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:18.160763    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f2d5a6bbcaf74c9380464df41b94313328f40d990b29bc04310c5a73cbf58d3"} err="failed to get container status \"0f2d5a6bbcaf74c9380464df41b94313328f40d990b29bc04310c5a73cbf58d3\": rpc error: code = NotFound desc = could not find container \"0f2d5a6bbcaf74c9380464df41b94313328f40d990b29bc04310c5a73cbf58d3\": container with ID starting with 0f2d5a6bbcaf74c9380464df41b94313328f40d990b29bc04310c5a73cbf58d3 not found: ID does not exist"
Nov 02 02:17:18.160884 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:18.160787    5875 scope.go:122] "RemoveContainer" containerID="0a8b7a7a2adfb721a57a171e3f36dff77027dc44d026150974e57e36da92d08b"
Nov 02 02:17:18.170423 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:18.170407    5875 scope.go:122] "RemoveContainer" containerID="0a8b7a7a2adfb721a57a171e3f36dff77027dc44d026150974e57e36da92d08b"
Nov 02 02:17:18.170687 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:17:18.170668    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a8b7a7a2adfb721a57a171e3f36dff77027dc44d026150974e57e36da92d08b\": container with ID starting with 0a8b7a7a2adfb721a57a171e3f36dff77027dc44d026150974e57e36da92d08b not found: ID does not exist" containerID="0a8b7a7a2adfb721a57a171e3f36dff77027dc44d026150974e57e36da92d08b"
Nov 02 02:17:18.170740 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:18.170695    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8b7a7a2adfb721a57a171e3f36dff77027dc44d026150974e57e36da92d08b"} err="failed to get container status \"0a8b7a7a2adfb721a57a171e3f36dff77027dc44d026150974e57e36da92d08b\": rpc error: code = NotFound desc = could not find container \"0a8b7a7a2adfb721a57a171e3f36dff77027dc44d026150974e57e36da92d08b\": container with ID starting with 0a8b7a7a2adfb721a57a171e3f36dff77027dc44d026150974e57e36da92d08b not found: ID does not exist"
Nov 02 02:17:18.313958 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:18.313911    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls"]
Nov 02 02:17:18.320037 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:18.320005    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4211c0-785c4dd895-9vxls"]
Nov 02 02:17:18.428908 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:18.428832    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm"]
Nov 02 02:17:18.432732 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:18.432706    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4211c0-5c78677f6f-xrrmm"]
Nov 02 02:17:18.693697 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:18.693620    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fa3f0ef-b33a-4fda-9e31-5393367463e2" path="/var/lib/kubelet/pods/0fa3f0ef-b33a-4fda-9e31-5393367463e2/volumes"
Nov 02 02:17:18.694015 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:18.694001    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad5288b9-b2da-41be-830a-72c877a15faf" path="/var/lib/kubelet/pods/ad5288b9-b2da-41be-830a-72c877a15faf/volumes"
Nov 02 02:17:23.170563 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:23.170529    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8"]
Nov 02 02:17:27.671939 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.671901    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db1f4f70-2qx5t"]
Nov 02 02:17:27.672454 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.672435    5875 cpu_manager.go:414] "containerMap: removing container" podUID="0fa3f0ef-b33a-4fda-9e31-5393367463e2" containerName="payload-processing"
Nov 02 02:17:27.672501 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.672455    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa3f0ef-b33a-4fda-9e31-5393367463e2" containerName="payload-processing"
Nov 02 02:17:27.672501 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.672476    5875 cpu_manager.go:414] "containerMap: removing container" podUID="ad5288b9-b2da-41be-830a-72c877a15faf" containerName="payload-pre-processing"
Nov 02 02:17:27.672501 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.672481    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad5288b9-b2da-41be-830a-72c877a15faf" containerName="payload-pre-processing"
Nov 02 02:17:27.672501 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.672498    5875 cpu_manager.go:414] "containerMap: removing container" podUID="e34877d3-dc45-4fd4-8d11-bb85c91458af" containerName="revoke-keys"
Nov 02 02:17:27.672685 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.672504    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e34877d3-dc45-4fd4-8d11-bb85c91458af" containerName="revoke-keys"
Nov 02 02:17:27.672685 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.672520    5875 cpu_manager.go:414] "containerMap: removing container" podUID="93d9b083-f014-4998-91b1-a90d3cce473c" containerName="istio-proxy"
Nov 02 02:17:27.672685 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.672526    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93d9b083-f014-4998-91b1-a90d3cce473c" containerName="istio-proxy"
Nov 02 02:17:27.672685 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.672541    5875 cpu_manager.go:414] "containerMap: removing container" podUID="93d9b083-f014-4998-91b1-a90d3cce473c" containerName="istio-proxy"
Nov 02 02:17:27.672685 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.672546    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93d9b083-f014-4998-91b1-a90d3cce473c" containerName="istio-proxy"
Nov 02 02:17:27.672685 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.672628    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="93d9b083-f014-4998-91b1-a90d3cce473c" containerName="istio-proxy"
Nov 02 02:17:27.672685 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.672636    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa3f0ef-b33a-4fda-9e31-5393367463e2" containerName="payload-processing"
Nov 02 02:17:27.672685 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.672643    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad5288b9-b2da-41be-830a-72c877a15faf" containerName="payload-pre-processing"
Nov 02 02:17:27.672685 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.672651    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="e34877d3-dc45-4fd4-8d11-bb85c91458af" containerName="revoke-keys"
Nov 02 02:17:27.675628 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.675613    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db1f4f70-2qx5t"
Nov 02 02:17:27.732009 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.731957    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:17:27.778561 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.778538    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:17:27.778716 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.778572    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:17:27.778716 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.778573    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:17:27.780567 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.780549    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db1f4f70-2qx5t"]
Nov 02 02:17:27.784135 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.784115    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/857ac035-c585-49ba-aec3-d6d9987b1d0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db1f4f70-2qx5t\" (UID: \"857ac035-c585-49ba-aec3-d6d9987b1d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db1f4f70-2qx5t"
Nov 02 02:17:27.784411 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.784385    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b65mk\" (UniqueName: \"kubernetes.io/projected/857ac035-c585-49ba-aec3-d6d9987b1d0e-kube-api-access-b65mk\") pod \"maas-api-revoke-sub-db1f4f70-2qx5t\" (UID: \"857ac035-c585-49ba-aec3-d6d9987b1d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db1f4f70-2qx5t"
Nov 02 02:17:27.784608 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.784585    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/857ac035-c585-49ba-aec3-d6d9987b1d0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db1f4f70-2qx5t\" (UID: \"857ac035-c585-49ba-aec3-d6d9987b1d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db1f4f70-2qx5t"
Nov 02 02:17:27.885437 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.885399    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b65mk\" (UniqueName: \"kubernetes.io/projected/857ac035-c585-49ba-aec3-d6d9987b1d0e-kube-api-access-b65mk\") pod \"maas-api-revoke-sub-db1f4f70-2qx5t\" (UID: \"857ac035-c585-49ba-aec3-d6d9987b1d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db1f4f70-2qx5t"
Nov 02 02:17:27.885629 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.885508    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/857ac035-c585-49ba-aec3-d6d9987b1d0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db1f4f70-2qx5t\" (UID: \"857ac035-c585-49ba-aec3-d6d9987b1d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db1f4f70-2qx5t"
Nov 02 02:17:27.885629 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.885536    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/857ac035-c585-49ba-aec3-d6d9987b1d0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db1f4f70-2qx5t\" (UID: \"857ac035-c585-49ba-aec3-d6d9987b1d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db1f4f70-2qx5t"
Nov 02 02:17:27.886195 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.886170    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/857ac035-c585-49ba-aec3-d6d9987b1d0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db1f4f70-2qx5t\" (UID: \"857ac035-c585-49ba-aec3-d6d9987b1d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db1f4f70-2qx5t"
Nov 02 02:17:27.887988 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.887969    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/857ac035-c585-49ba-aec3-d6d9987b1d0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db1f4f70-2qx5t\" (UID: \"857ac035-c585-49ba-aec3-d6d9987b1d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db1f4f70-2qx5t"
Nov 02 02:17:27.946300 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.946239    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b65mk\" (UniqueName: \"kubernetes.io/projected/857ac035-c585-49ba-aec3-d6d9987b1d0e-kube-api-access-b65mk\") pod \"maas-api-revoke-sub-db1f4f70-2qx5t\" (UID: \"857ac035-c585-49ba-aec3-d6d9987b1d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db1f4f70-2qx5t"
Nov 02 02:17:27.988741 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:27.988711    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db1f4f70-2qx5t"
Nov 02 02:17:28.171208 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:28.171169    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8" podUID="450879f9-c443-4a35-a13d-6872b0294337" containerName="payload-processing" containerID="cri-o://a5f046ab3d2daf6d0c5aee11e6a6b80ec0a2a733e4d4e5a8377c888bb54b37b0" gracePeriod=25
Nov 02 02:17:28.360888 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:28.360863    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db1f4f70-2qx5t"]
Nov 02 02:17:28.363009 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:17:28.362971    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod857ac035_c585_49ba_aec3_d6d9987b1d0e.slice/crio-f4d4327c3677edad13d7bea36e8f7e5224296384b0c9b70684267b3fd8246b68 WatchSource:0}: Error finding container f4d4327c3677edad13d7bea36e8f7e5224296384b0c9b70684267b3fd8246b68: Status 404 returned error can't find the container with id f4d4327c3677edad13d7bea36e8f7e5224296384b0c9b70684267b3fd8246b68
Nov 02 02:17:29.223780 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:29.223750    5875 generic.go:356] "Generic (PLEG): container finished" podID="857ac035-c585-49ba-aec3-d6d9987b1d0e" containerID="238743f0dc145478d7262b2b9c7c63a7661ce3ce55aef562f42aeeb4d61f62a8" exitCode=0
Nov 02 02:17:29.224080 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:29.223840    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db1f4f70-2qx5t" event={"ID":"857ac035-c585-49ba-aec3-d6d9987b1d0e","Type":"ContainerDied","Data":"238743f0dc145478d7262b2b9c7c63a7661ce3ce55aef562f42aeeb4d61f62a8"}
Nov 02 02:17:29.224080 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:29.223891    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db1f4f70-2qx5t" event={"ID":"857ac035-c585-49ba-aec3-d6d9987b1d0e","Type":"ContainerStarted","Data":"f4d4327c3677edad13d7bea36e8f7e5224296384b0c9b70684267b3fd8246b68"}
Nov 02 02:17:30.048418 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:30.048379    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aab23e3f-lgsvs"]
Nov 02 02:17:30.096049 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:30.096019    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aab23e3f-lgsvs"]
Nov 02 02:17:30.375521 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:30.375498    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db1f4f70-2qx5t"
Nov 02 02:17:30.512144 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:30.512111    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/857ac035-c585-49ba-aec3-d6d9987b1d0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/857ac035-c585-49ba-aec3-d6d9987b1d0e-maas-api-xks-ca\") pod \"857ac035-c585-49ba-aec3-d6d9987b1d0e\" (UID: \"857ac035-c585-49ba-aec3-d6d9987b1d0e\") "
Nov 02 02:17:30.512431 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:30.512210    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/857ac035-c585-49ba-aec3-d6d9987b1d0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/857ac035-c585-49ba-aec3-d6d9987b1d0e-maas-api-openshift-ca\") pod \"857ac035-c585-49ba-aec3-d6d9987b1d0e\" (UID: \"857ac035-c585-49ba-aec3-d6d9987b1d0e\") "
Nov 02 02:17:30.512431 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:30.512357    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/857ac035-c585-49ba-aec3-d6d9987b1d0e-kube-api-access-b65mk\" (UniqueName: \"kubernetes.io/projected/857ac035-c585-49ba-aec3-d6d9987b1d0e-kube-api-access-b65mk\") pod \"857ac035-c585-49ba-aec3-d6d9987b1d0e\" (UID: \"857ac035-c585-49ba-aec3-d6d9987b1d0e\") "
Nov 02 02:17:30.512621 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:30.512590    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/857ac035-c585-49ba-aec3-d6d9987b1d0e-maas-api-openshift-ca" pod "857ac035-c585-49ba-aec3-d6d9987b1d0e" (UID: "857ac035-c585-49ba-aec3-d6d9987b1d0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:17:30.512800 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:30.512781    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/857ac035-c585-49ba-aec3-d6d9987b1d0e-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:30.514525 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:30.514503    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/857ac035-c585-49ba-aec3-d6d9987b1d0e-kube-api-access-b65mk" pod "857ac035-c585-49ba-aec3-d6d9987b1d0e" (UID: "857ac035-c585-49ba-aec3-d6d9987b1d0e"). InnerVolumeSpecName "kube-api-access-b65mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:17:30.514624 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:30.514563    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/857ac035-c585-49ba-aec3-d6d9987b1d0e-maas-api-xks-ca" pod "857ac035-c585-49ba-aec3-d6d9987b1d0e" (UID: "857ac035-c585-49ba-aec3-d6d9987b1d0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:17:30.613910 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:30.613840    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/857ac035-c585-49ba-aec3-d6d9987b1d0e-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:30.613910 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:30.613867    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b65mk\" (UniqueName: \"kubernetes.io/projected/857ac035-c585-49ba-aec3-d6d9987b1d0e-kube-api-access-b65mk\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:30.700607 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:30.700575    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34647927-454c-4bb5-8c38-70c2502839e7" path="/var/lib/kubelet/pods/34647927-454c-4bb5-8c38-70c2502839e7/volumes"
Nov 02 02:17:31.237327 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:31.237297    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db1f4f70-2qx5t"
Nov 02 02:17:31.237531 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:31.237326    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db1f4f70-2qx5t" event={"ID":"857ac035-c585-49ba-aec3-d6d9987b1d0e","Type":"ContainerDied","Data":"f4d4327c3677edad13d7bea36e8f7e5224296384b0c9b70684267b3fd8246b68"}
Nov 02 02:17:31.237531 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:31.237379    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4d4327c3677edad13d7bea36e8f7e5224296384b0c9b70684267b3fd8246b68"
Nov 02 02:17:37.897787 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:37.897748    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8" podUID="450879f9-c443-4a35-a13d-6872b0294337" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:17:39.110100 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:39.110070    5875 scope.go:122] "RemoveContainer" containerID="2d4a0f53f9f590571cb40ea5526d36121cb41ff2ff53ee5f90408a278a2ba596"
Nov 02 02:17:41.318568 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:17:41.318469    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:17:44.714814 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:44.714779    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:17:45.112738 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:45.112665    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9"]
Nov 02 02:17:45.162369 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:45.162335    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-0ebdb3-4dkv9"]
Nov 02 02:17:46.693448 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:46.693418    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e34877d3-dc45-4fd4-8d11-bb85c91458af" path="/var/lib/kubelet/pods/e34877d3-dc45-4fd4-8d11-bb85c91458af/volumes"
Nov 02 02:17:47.897961 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:47.897928    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8" podUID="450879f9-c443-4a35-a13d-6872b0294337" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 02:17:48.336817 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:48.336785    5875 generic.go:356] "Generic (PLEG): container finished" podID="450879f9-c443-4a35-a13d-6872b0294337" containerID="a5f046ab3d2daf6d0c5aee11e6a6b80ec0a2a733e4d4e5a8377c888bb54b37b0" exitCode=0
Nov 02 02:17:48.337009 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:48.336860    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8" event={"ID":"450879f9-c443-4a35-a13d-6872b0294337","Type":"ContainerDied","Data":"a5f046ab3d2daf6d0c5aee11e6a6b80ec0a2a733e4d4e5a8377c888bb54b37b0"}
Nov 02 02:17:48.424674 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:48.424647    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8"
Nov 02 02:17:48.493573 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:48.493546    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/450879f9-c443-4a35-a13d-6872b0294337-kube-api-access-tc42f\" (UniqueName: \"kubernetes.io/projected/450879f9-c443-4a35-a13d-6872b0294337-kube-api-access-tc42f\") pod \"450879f9-c443-4a35-a13d-6872b0294337\" (UID: \"450879f9-c443-4a35-a13d-6872b0294337\") "
Nov 02 02:17:48.493711 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:48.493676    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/450879f9-c443-4a35-a13d-6872b0294337-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/450879f9-c443-4a35-a13d-6872b0294337-plugins-config-volume\") pod \"450879f9-c443-4a35-a13d-6872b0294337\" (UID: \"450879f9-c443-4a35-a13d-6872b0294337\") "
Nov 02 02:17:48.493996 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:48.493974    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/450879f9-c443-4a35-a13d-6872b0294337-plugins-config-volume" pod "450879f9-c443-4a35-a13d-6872b0294337" (UID: "450879f9-c443-4a35-a13d-6872b0294337"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:17:48.495699 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:48.495673    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/450879f9-c443-4a35-a13d-6872b0294337-kube-api-access-tc42f" pod "450879f9-c443-4a35-a13d-6872b0294337" (UID: "450879f9-c443-4a35-a13d-6872b0294337"). InnerVolumeSpecName "kube-api-access-tc42f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:17:48.595350 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:48.595315    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/450879f9-c443-4a35-a13d-6872b0294337-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:48.595350 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:48.595343    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc42f\" (UniqueName: \"kubernetes.io/projected/450879f9-c443-4a35-a13d-6872b0294337-kube-api-access-tc42f\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:49.343764 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:49.343732    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8"
Nov 02 02:17:49.344203 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:49.343738    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8" event={"ID":"450879f9-c443-4a35-a13d-6872b0294337","Type":"ContainerDied","Data":"c4a973270438ed2d2c23c5481f687967102924b3fd0ce403d397fcd86f1fc2ec"}
Nov 02 02:17:49.344203 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:49.343870    5875 scope.go:122] "RemoveContainer" containerID="a5f046ab3d2daf6d0c5aee11e6a6b80ec0a2a733e4d4e5a8377c888bb54b37b0"
Nov 02 02:17:49.507704 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:49.507669    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8"]
Nov 02 02:17:49.557269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:49.557237    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-0ebdb3-6fd545cf67-x42g8"]
Nov 02 02:17:50.693510 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:50.693479    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="450879f9-c443-4a35-a13d-6872b0294337" path="/var/lib/kubelet/pods/450879f9-c443-4a35-a13d-6872b0294337/volumes"
Nov 02 02:17:58.456125 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:58.456085    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f8c46b7cf-qn58p"]
Nov 02 02:17:58.456587 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:58.456317    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5f8c46b7cf-qn58p" podUID="f03055d8-42b7-4c1b-9900-4c891e0ac145" containerName="manager" containerID="cri-o://c4f88c6047525a6e8e8ab16f42258af2f67077aa2fdcd3ee4cc31fcee6c5f288" gracePeriod=30
Nov 02 02:17:58.712914 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:58.712860    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f8c46b7cf-qn58p"
Nov 02 02:17:58.802216 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:58.802178    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f03055d8-42b7-4c1b-9900-4c891e0ac145-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f03055d8-42b7-4c1b-9900-4c891e0ac145-webhook-cert\") pod \"f03055d8-42b7-4c1b-9900-4c891e0ac145\" (UID: \"f03055d8-42b7-4c1b-9900-4c891e0ac145\") "
Nov 02 02:17:58.802378 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:58.802331    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f03055d8-42b7-4c1b-9900-4c891e0ac145-kube-api-access-tql7t\" (UniqueName: \"kubernetes.io/projected/f03055d8-42b7-4c1b-9900-4c891e0ac145-kube-api-access-tql7t\") pod \"f03055d8-42b7-4c1b-9900-4c891e0ac145\" (UID: \"f03055d8-42b7-4c1b-9900-4c891e0ac145\") "
Nov 02 02:17:58.802378 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:58.802369    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f03055d8-42b7-4c1b-9900-4c891e0ac145-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f03055d8-42b7-4c1b-9900-4c891e0ac145-metrics-certs\") pod \"f03055d8-42b7-4c1b-9900-4c891e0ac145\" (UID: \"f03055d8-42b7-4c1b-9900-4c891e0ac145\") "
Nov 02 02:17:58.804391 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:58.804361    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f03055d8-42b7-4c1b-9900-4c891e0ac145-metrics-certs" pod "f03055d8-42b7-4c1b-9900-4c891e0ac145" (UID: "f03055d8-42b7-4c1b-9900-4c891e0ac145"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:17:58.804497 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:58.804433    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03055d8-42b7-4c1b-9900-4c891e0ac145-kube-api-access-tql7t" pod "f03055d8-42b7-4c1b-9900-4c891e0ac145" (UID: "f03055d8-42b7-4c1b-9900-4c891e0ac145"). InnerVolumeSpecName "kube-api-access-tql7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:17:58.804497 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:58.804443    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f03055d8-42b7-4c1b-9900-4c891e0ac145-webhook-cert" pod "f03055d8-42b7-4c1b-9900-4c891e0ac145" (UID: "f03055d8-42b7-4c1b-9900-4c891e0ac145"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:17:58.903642 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:58.903612    5875 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f03055d8-42b7-4c1b-9900-4c891e0ac145-metrics-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:58.903642 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:58.903638    5875 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f03055d8-42b7-4c1b-9900-4c891e0ac145-webhook-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:58.903642 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:58.903648    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tql7t\" (UniqueName: \"kubernetes.io/projected/f03055d8-42b7-4c1b-9900-4c891e0ac145-kube-api-access-tql7t\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:17:59.401745 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:59.401708    5875 generic.go:356] "Generic (PLEG): container finished" podID="f03055d8-42b7-4c1b-9900-4c891e0ac145" containerID="c4f88c6047525a6e8e8ab16f42258af2f67077aa2fdcd3ee4cc31fcee6c5f288" exitCode=0
Nov 02 02:17:59.401930 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:59.401763    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f8c46b7cf-qn58p" event={"ID":"f03055d8-42b7-4c1b-9900-4c891e0ac145","Type":"ContainerDied","Data":"c4f88c6047525a6e8e8ab16f42258af2f67077aa2fdcd3ee4cc31fcee6c5f288"}
Nov 02 02:17:59.401930 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:59.401769    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f8c46b7cf-qn58p"
Nov 02 02:17:59.401930 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:59.401805    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f8c46b7cf-qn58p" event={"ID":"f03055d8-42b7-4c1b-9900-4c891e0ac145","Type":"ContainerDied","Data":"a6dfef63aad857793530ae12265990fe06de8edaecc9383161acb2f5c75aa485"}
Nov 02 02:17:59.401930 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:59.401827    5875 scope.go:122] "RemoveContainer" containerID="c4f88c6047525a6e8e8ab16f42258af2f67077aa2fdcd3ee4cc31fcee6c5f288"
Nov 02 02:17:59.412647 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:59.412627    5875 scope.go:122] "RemoveContainer" containerID="c4f88c6047525a6e8e8ab16f42258af2f67077aa2fdcd3ee4cc31fcee6c5f288"
Nov 02 02:17:59.412950 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:17:59.412929    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f88c6047525a6e8e8ab16f42258af2f67077aa2fdcd3ee4cc31fcee6c5f288\": container with ID starting with c4f88c6047525a6e8e8ab16f42258af2f67077aa2fdcd3ee4cc31fcee6c5f288 not found: ID does not exist" containerID="c4f88c6047525a6e8e8ab16f42258af2f67077aa2fdcd3ee4cc31fcee6c5f288"
Nov 02 02:17:59.413024 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:59.412962    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f88c6047525a6e8e8ab16f42258af2f67077aa2fdcd3ee4cc31fcee6c5f288"} err="failed to get container status \"c4f88c6047525a6e8e8ab16f42258af2f67077aa2fdcd3ee4cc31fcee6c5f288\": rpc error: code = NotFound desc = could not find container \"c4f88c6047525a6e8e8ab16f42258af2f67077aa2fdcd3ee4cc31fcee6c5f288\": container with ID starting with c4f88c6047525a6e8e8ab16f42258af2f67077aa2fdcd3ee4cc31fcee6c5f288 not found: ID does not exist"
Nov 02 02:17:59.566282 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:59.566248    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f8c46b7cf-qn58p"]
Nov 02 02:17:59.615824 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:17:59.615789    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5f8c46b7cf-qn58p"]
Nov 02 02:18:00.373806 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.373773    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:18:00.460294 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.460262    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5f8c46b7cf-zffd5"]
Nov 02 02:18:00.460689 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.460677    5875 cpu_manager.go:414] "containerMap: removing container" podUID="f03055d8-42b7-4c1b-9900-4c891e0ac145" containerName="manager"
Nov 02 02:18:00.460740 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.460691    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f03055d8-42b7-4c1b-9900-4c891e0ac145" containerName="manager"
Nov 02 02:18:00.460740 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.460719    5875 cpu_manager.go:414] "containerMap: removing container" podUID="857ac035-c585-49ba-aec3-d6d9987b1d0e" containerName="revoke-keys"
Nov 02 02:18:00.460740 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.460725    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="857ac035-c585-49ba-aec3-d6d9987b1d0e" containerName="revoke-keys"
Nov 02 02:18:00.460740 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.460740    5875 cpu_manager.go:414] "containerMap: removing container" podUID="450879f9-c443-4a35-a13d-6872b0294337" containerName="payload-processing"
Nov 02 02:18:00.460876 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.460745    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="450879f9-c443-4a35-a13d-6872b0294337" containerName="payload-processing"
Nov 02 02:18:00.460876 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.460814    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="450879f9-c443-4a35-a13d-6872b0294337" containerName="payload-processing"
Nov 02 02:18:00.460876 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.460826    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="f03055d8-42b7-4c1b-9900-4c891e0ac145" containerName="manager"
Nov 02 02:18:00.460876 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.460834    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="93d9b083-f014-4998-91b1-a90d3cce473c" containerName="istio-proxy"
Nov 02 02:18:00.460876 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.460843    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="857ac035-c585-49ba-aec3-d6d9987b1d0e" containerName="revoke-keys"
Nov 02 02:18:00.465444 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.465425    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f8c46b7cf-zffd5"
Nov 02 02:18:00.517614 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.517586    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 02:18:00.517614 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.517606    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vj2vs\""
Nov 02 02:18:00.517614 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.517599    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 02:18:00.517839 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.517736    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 02:18:00.566699 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.566659    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f8c46b7cf-zffd5"]
Nov 02 02:18:00.620700 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.620672    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb94q\" (UniqueName: \"kubernetes.io/projected/aa38997f-86ee-45d2-9ad2-60cf5e6c529f-kube-api-access-wb94q\") pod \"maas-controller-5f8c46b7cf-zffd5\" (UID: \"aa38997f-86ee-45d2-9ad2-60cf5e6c529f\") " pod="opendatahub/maas-controller-5f8c46b7cf-zffd5"
Nov 02 02:18:00.620847 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.620710    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa38997f-86ee-45d2-9ad2-60cf5e6c529f-webhook-cert\") pod \"maas-controller-5f8c46b7cf-zffd5\" (UID: \"aa38997f-86ee-45d2-9ad2-60cf5e6c529f\") " pod="opendatahub/maas-controller-5f8c46b7cf-zffd5"
Nov 02 02:18:00.620847 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.620729    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa38997f-86ee-45d2-9ad2-60cf5e6c529f-metrics-certs\") pod \"maas-controller-5f8c46b7cf-zffd5\" (UID: \"aa38997f-86ee-45d2-9ad2-60cf5e6c529f\") " pod="opendatahub/maas-controller-5f8c46b7cf-zffd5"
Nov 02 02:18:00.712424 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.712392    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f03055d8-42b7-4c1b-9900-4c891e0ac145" path="/var/lib/kubelet/pods/f03055d8-42b7-4c1b-9900-4c891e0ac145/volumes"
Nov 02 02:18:00.721795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.721764    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb94q\" (UniqueName: \"kubernetes.io/projected/aa38997f-86ee-45d2-9ad2-60cf5e6c529f-kube-api-access-wb94q\") pod \"maas-controller-5f8c46b7cf-zffd5\" (UID: \"aa38997f-86ee-45d2-9ad2-60cf5e6c529f\") " pod="opendatahub/maas-controller-5f8c46b7cf-zffd5"
Nov 02 02:18:00.721935 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.721822    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa38997f-86ee-45d2-9ad2-60cf5e6c529f-webhook-cert\") pod \"maas-controller-5f8c46b7cf-zffd5\" (UID: \"aa38997f-86ee-45d2-9ad2-60cf5e6c529f\") " pod="opendatahub/maas-controller-5f8c46b7cf-zffd5"
Nov 02 02:18:00.721935 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.721854    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa38997f-86ee-45d2-9ad2-60cf5e6c529f-metrics-certs\") pod \"maas-controller-5f8c46b7cf-zffd5\" (UID: \"aa38997f-86ee-45d2-9ad2-60cf5e6c529f\") " pod="opendatahub/maas-controller-5f8c46b7cf-zffd5"
Nov 02 02:18:00.725258 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.725211    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa38997f-86ee-45d2-9ad2-60cf5e6c529f-webhook-cert\") pod \"maas-controller-5f8c46b7cf-zffd5\" (UID: \"aa38997f-86ee-45d2-9ad2-60cf5e6c529f\") " pod="opendatahub/maas-controller-5f8c46b7cf-zffd5"
Nov 02 02:18:00.725653 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.725635    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa38997f-86ee-45d2-9ad2-60cf5e6c529f-metrics-certs\") pod \"maas-controller-5f8c46b7cf-zffd5\" (UID: \"aa38997f-86ee-45d2-9ad2-60cf5e6c529f\") " pod="opendatahub/maas-controller-5f8c46b7cf-zffd5"
Nov 02 02:18:00.784036 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:00.783989    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb94q\" (UniqueName: \"kubernetes.io/projected/aa38997f-86ee-45d2-9ad2-60cf5e6c529f-kube-api-access-wb94q\") pod \"maas-controller-5f8c46b7cf-zffd5\" (UID: \"aa38997f-86ee-45d2-9ad2-60cf5e6c529f\") " pod="opendatahub/maas-controller-5f8c46b7cf-zffd5"
Nov 02 02:18:01.078845 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:01.078751    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f8c46b7cf-zffd5"
Nov 02 02:18:01.481565 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:01.481525    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f8c46b7cf-zffd5"]
Nov 02 02:18:01.486788 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:18:01.486762    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa38997f_86ee_45d2_9ad2_60cf5e6c529f.slice/crio-7bccec2ca42088ff63f926f1893486e1d80c26aaab20b4f63a26ce5a51c3171b WatchSource:0}: Error finding container 7bccec2ca42088ff63f926f1893486e1d80c26aaab20b4f63a26ce5a51c3171b: Status 404 returned error can't find the container with id 7bccec2ca42088ff63f926f1893486e1d80c26aaab20b4f63a26ce5a51c3171b
Nov 02 02:18:01.488865 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:01.488843    5875 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 02:18:02.461901 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:02.461846    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f8c46b7cf-zffd5" event={"ID":"aa38997f-86ee-45d2-9ad2-60cf5e6c529f","Type":"ContainerStarted","Data":"7bccec2ca42088ff63f926f1893486e1d80c26aaab20b4f63a26ce5a51c3171b"}
Nov 02 02:18:03.472247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:03.472066    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f8c46b7cf-zffd5" event={"ID":"aa38997f-86ee-45d2-9ad2-60cf5e6c529f","Type":"ContainerStarted","Data":"9779af49ecd5c525070674d582eb8737faf01c7e7e5420e057c2443a144f4c43"}
Nov 02 02:18:03.472247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:03.472130    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5f8c46b7cf-zffd5"
Nov 02 02:18:03.590385 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:03.590324    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5f8c46b7cf-zffd5" podStartSLOduration=2.54002181 podStartE2EDuration="3.590308588s" totalImagesPullingTime="1.050286778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:18:03.589568442 +0000 UTC m=+2731.030604940" watchObservedRunningTime="2026-11-02 02:18:03.590308588 +0000 UTC m=+2731.031345081"
Nov 02 02:18:09.048467 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:09.048427    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aa3a765-qhw9f"]
Nov 02 02:18:09.064966 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:09.064876    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83674a01-9srq4"]
Nov 02 02:18:09.098344 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:09.098313    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aa3a765-qhw9f"]
Nov 02 02:18:09.099886 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:09.099865    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83674a01-9srq4"]
Nov 02 02:18:10.693244 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:10.693186    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae" path="/var/lib/kubelet/pods/3a1cd1ad-cbd0-4d82-beb1-f6dc1769caae/volumes"
Nov 02 02:18:10.693792 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:10.693772    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e53ad90e-2453-491f-a47f-e9f25ce72b18" path="/var/lib/kubelet/pods/e53ad90e-2453-491f-a47f-e9f25ce72b18/volumes"
Nov 02 02:18:11.318326 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:18:11.318292    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:18:14.480316 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:14.480282    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5f8c46b7cf-zffd5"
Nov 02 02:18:18.045718 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:18.045680    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a02c918e-x6797"]
Nov 02 02:18:18.095480 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:18.095444    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a02c918e-x6797"]
Nov 02 02:18:18.692767 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:18.692735    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e7218bf-2867-4f6d-bdb1-67678e906ea0" path="/var/lib/kubelet/pods/2e7218bf-2867-4f6d-bdb1-67678e906ea0/volumes"
Nov 02 02:18:19.047572 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:19.047536    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9635733-b95tr"]
Nov 02 02:18:19.096352 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:19.096320    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9635733-b95tr"]
Nov 02 02:18:20.693441 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:20.693408    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ebb7498-7d43-456b-a98c-8559a858db6a" path="/var/lib/kubelet/pods/6ebb7498-7d43-456b-a98c-8559a858db6a/volumes"
Nov 02 02:18:34.433132 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:34.433094    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f8c46b7cf-zffd5"]
Nov 02 02:18:34.433736 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:34.433365    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5f8c46b7cf-zffd5" podUID="aa38997f-86ee-45d2-9ad2-60cf5e6c529f" containerName="manager" containerID="cri-o://9779af49ecd5c525070674d582eb8737faf01c7e7e5420e057c2443a144f4c43" gracePeriod=30
Nov 02 02:18:34.658532 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:34.658501    5875 generic.go:356] "Generic (PLEG): container finished" podID="aa38997f-86ee-45d2-9ad2-60cf5e6c529f" containerID="9779af49ecd5c525070674d582eb8737faf01c7e7e5420e057c2443a144f4c43" exitCode=0
Nov 02 02:18:34.658736 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:34.658585    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f8c46b7cf-zffd5" event={"ID":"aa38997f-86ee-45d2-9ad2-60cf5e6c529f","Type":"ContainerDied","Data":"9779af49ecd5c525070674d582eb8737faf01c7e7e5420e057c2443a144f4c43"}
Nov 02 02:18:34.688518 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:34.688463    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f8c46b7cf-zffd5"
Nov 02 02:18:34.874237 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:34.874185    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa38997f-86ee-45d2-9ad2-60cf5e6c529f-kube-api-access-wb94q\" (UniqueName: \"kubernetes.io/projected/aa38997f-86ee-45d2-9ad2-60cf5e6c529f-kube-api-access-wb94q\") pod \"aa38997f-86ee-45d2-9ad2-60cf5e6c529f\" (UID: \"aa38997f-86ee-45d2-9ad2-60cf5e6c529f\") "
Nov 02 02:18:34.874434 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:34.874292    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa38997f-86ee-45d2-9ad2-60cf5e6c529f-webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa38997f-86ee-45d2-9ad2-60cf5e6c529f-webhook-cert\") pod \"aa38997f-86ee-45d2-9ad2-60cf5e6c529f\" (UID: \"aa38997f-86ee-45d2-9ad2-60cf5e6c529f\") "
Nov 02 02:18:34.874434 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:34.874319    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa38997f-86ee-45d2-9ad2-60cf5e6c529f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa38997f-86ee-45d2-9ad2-60cf5e6c529f-metrics-certs\") pod \"aa38997f-86ee-45d2-9ad2-60cf5e6c529f\" (UID: \"aa38997f-86ee-45d2-9ad2-60cf5e6c529f\") "
Nov 02 02:18:34.876674 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:34.876638    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa38997f-86ee-45d2-9ad2-60cf5e6c529f-webhook-cert" pod "aa38997f-86ee-45d2-9ad2-60cf5e6c529f" (UID: "aa38997f-86ee-45d2-9ad2-60cf5e6c529f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:18:34.876674 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:34.876658    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa38997f-86ee-45d2-9ad2-60cf5e6c529f-metrics-certs" pod "aa38997f-86ee-45d2-9ad2-60cf5e6c529f" (UID: "aa38997f-86ee-45d2-9ad2-60cf5e6c529f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:18:34.876881 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:34.876697    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa38997f-86ee-45d2-9ad2-60cf5e6c529f-kube-api-access-wb94q" pod "aa38997f-86ee-45d2-9ad2-60cf5e6c529f" (UID: "aa38997f-86ee-45d2-9ad2-60cf5e6c529f"). InnerVolumeSpecName "kube-api-access-wb94q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:18:34.975254 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:34.975204    5875 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa38997f-86ee-45d2-9ad2-60cf5e6c529f-webhook-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:18:34.975254 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:34.975251    5875 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa38997f-86ee-45d2-9ad2-60cf5e6c529f-metrics-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:18:34.975459 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:34.975262    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb94q\" (UniqueName: \"kubernetes.io/projected/aa38997f-86ee-45d2-9ad2-60cf5e6c529f-kube-api-access-wb94q\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:18:35.665293 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:35.665261    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f8c46b7cf-zffd5"
Nov 02 02:18:35.665293 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:35.665279    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f8c46b7cf-zffd5" event={"ID":"aa38997f-86ee-45d2-9ad2-60cf5e6c529f","Type":"ContainerDied","Data":"7bccec2ca42088ff63f926f1893486e1d80c26aaab20b4f63a26ce5a51c3171b"}
Nov 02 02:18:35.665880 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:35.665343    5875 scope.go:122] "RemoveContainer" containerID="9779af49ecd5c525070674d582eb8737faf01c7e7e5420e057c2443a144f4c43"
Nov 02 02:18:35.828714 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:35.828684    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f8c46b7cf-zffd5"]
Nov 02 02:18:35.878326 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:35.878292    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5f8c46b7cf-zffd5"]
Nov 02 02:18:36.693204 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:36.693168    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa38997f-86ee-45d2-9ad2-60cf5e6c529f" path="/var/lib/kubelet/pods/aa38997f-86ee-45d2-9ad2-60cf5e6c529f/volumes"
Nov 02 02:18:37.184867 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:37.184837    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5f8c46b7cf-5584m"]
Nov 02 02:18:37.185297 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:37.185283    5875 cpu_manager.go:414] "containerMap: removing container" podUID="aa38997f-86ee-45d2-9ad2-60cf5e6c529f" containerName="manager"
Nov 02 02:18:37.185354 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:37.185298    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa38997f-86ee-45d2-9ad2-60cf5e6c529f" containerName="manager"
Nov 02 02:18:37.185410 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:37.185402    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa38997f-86ee-45d2-9ad2-60cf5e6c529f" containerName="manager"
Nov 02 02:18:37.189727 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:37.189708    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f8c46b7cf-5584m"
Nov 02 02:18:37.291211 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:37.291171    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 02:18:37.291416 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:37.291363    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 02:18:37.291416 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:37.291378    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 02:18:37.291416 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:37.291410    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vj2vs\""
Nov 02 02:18:37.293315 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:37.293291    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f8c46b7cf-5584m"]
Nov 02 02:18:37.297589 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:37.297566    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ba1716a-8773-45a5-902e-9098578180bf-webhook-cert\") pod \"maas-controller-5f8c46b7cf-5584m\" (UID: \"6ba1716a-8773-45a5-902e-9098578180bf\") " pod="opendatahub/maas-controller-5f8c46b7cf-5584m"
Nov 02 02:18:37.297696 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:37.297679    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk9b2\" (UniqueName: \"kubernetes.io/projected/6ba1716a-8773-45a5-902e-9098578180bf-kube-api-access-bk9b2\") pod \"maas-controller-5f8c46b7cf-5584m\" (UID: \"6ba1716a-8773-45a5-902e-9098578180bf\") " pod="opendatahub/maas-controller-5f8c46b7cf-5584m"
Nov 02 02:18:37.297755 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:37.297742    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ba1716a-8773-45a5-902e-9098578180bf-metrics-certs\") pod \"maas-controller-5f8c46b7cf-5584m\" (UID: \"6ba1716a-8773-45a5-902e-9098578180bf\") " pod="opendatahub/maas-controller-5f8c46b7cf-5584m"
Nov 02 02:18:37.399294 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:37.399262    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ba1716a-8773-45a5-902e-9098578180bf-metrics-certs\") pod \"maas-controller-5f8c46b7cf-5584m\" (UID: \"6ba1716a-8773-45a5-902e-9098578180bf\") " pod="opendatahub/maas-controller-5f8c46b7cf-5584m"
Nov 02 02:18:37.399472 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:37.399351    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ba1716a-8773-45a5-902e-9098578180bf-webhook-cert\") pod \"maas-controller-5f8c46b7cf-5584m\" (UID: \"6ba1716a-8773-45a5-902e-9098578180bf\") " pod="opendatahub/maas-controller-5f8c46b7cf-5584m"
Nov 02 02:18:37.399472 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:37.399397    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk9b2\" (UniqueName: \"kubernetes.io/projected/6ba1716a-8773-45a5-902e-9098578180bf-kube-api-access-bk9b2\") pod \"maas-controller-5f8c46b7cf-5584m\" (UID: \"6ba1716a-8773-45a5-902e-9098578180bf\") " pod="opendatahub/maas-controller-5f8c46b7cf-5584m"
Nov 02 02:18:37.401896 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:37.401869    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ba1716a-8773-45a5-902e-9098578180bf-metrics-certs\") pod \"maas-controller-5f8c46b7cf-5584m\" (UID: \"6ba1716a-8773-45a5-902e-9098578180bf\") " pod="opendatahub/maas-controller-5f8c46b7cf-5584m"
Nov 02 02:18:37.402020 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:37.401901    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ba1716a-8773-45a5-902e-9098578180bf-webhook-cert\") pod \"maas-controller-5f8c46b7cf-5584m\" (UID: \"6ba1716a-8773-45a5-902e-9098578180bf\") " pod="opendatahub/maas-controller-5f8c46b7cf-5584m"
Nov 02 02:18:37.460633 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:37.460597    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk9b2\" (UniqueName: \"kubernetes.io/projected/6ba1716a-8773-45a5-902e-9098578180bf-kube-api-access-bk9b2\") pod \"maas-controller-5f8c46b7cf-5584m\" (UID: \"6ba1716a-8773-45a5-902e-9098578180bf\") " pod="opendatahub/maas-controller-5f8c46b7cf-5584m"
Nov 02 02:18:37.503149 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:37.503123    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f8c46b7cf-5584m"
Nov 02 02:18:37.879203 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:37.879178    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f8c46b7cf-5584m"]
Nov 02 02:18:37.881879 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:18:37.881851    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba1716a_8773_45a5_902e_9098578180bf.slice/crio-e9e68d2b30a941776eef71bf25fc2a1fd259f52528ef62e4a4d017b450d51333 WatchSource:0}: Error finding container e9e68d2b30a941776eef71bf25fc2a1fd259f52528ef62e4a4d017b450d51333: Status 404 returned error can't find the container with id e9e68d2b30a941776eef71bf25fc2a1fd259f52528ef62e4a4d017b450d51333
Nov 02 02:18:38.684115 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:38.684085    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f8c46b7cf-5584m" event={"ID":"6ba1716a-8773-45a5-902e-9098578180bf","Type":"ContainerStarted","Data":"e9e68d2b30a941776eef71bf25fc2a1fd259f52528ef62e4a4d017b450d51333"}
Nov 02 02:18:39.148580 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:39.148544    5875 scope.go:122] "RemoveContainer" containerID="29c493e5452b5dffc7b7bf20fe44fee30d3e8550582be8a75737f874b4e78eb6"
Nov 02 02:18:39.157865 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:39.157843    5875 scope.go:122] "RemoveContainer" containerID="1d4ccf17b1f934bdd4358ee3c2201ace5745857905b2279c3fc7b0b13f5305a5"
Nov 02 02:18:39.166582 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:39.166563    5875 scope.go:122] "RemoveContainer" containerID="6546f60fc70f6cd9004944ec1db41f61fe71aeaf667f46a0dd184c0b38fd7540"
Nov 02 02:18:39.174858 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:39.174838    5875 scope.go:122] "RemoveContainer" containerID="1943be5bbb0f2e77c63fb9dadd986b2623d9a8a96366f148b074ee010a1943ec"
Nov 02 02:18:39.691871 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:39.691838    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f8c46b7cf-5584m" event={"ID":"6ba1716a-8773-45a5-902e-9098578180bf","Type":"ContainerStarted","Data":"749a04020f487bfc8f1ed6ab75e13b46f3787127e5695e64dbcbbc4c9a1d8d23"}
Nov 02 02:18:39.692069 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:39.691932    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5f8c46b7cf-5584m"
Nov 02 02:18:39.809959 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:39.809880    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5f8c46b7cf-5584m" podStartSLOduration=1.944010938 podStartE2EDuration="2.809836556s" totalImagesPullingTime="865.825618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:18:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:18:39.809377011 +0000 UTC m=+2767.250413511" watchObservedRunningTime="2026-11-02 02:18:39.809836556 +0000 UTC m=+2767.250873056"
Nov 02 02:18:41.316489 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:18:41.316456    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:18:50.698504 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:18:50.698476    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5f8c46b7cf-5584m"
Nov 02 02:19:11.321445 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:19:11.321349    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:19:22.668949 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:19:22.668915    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:19:39.221915 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:19:39.221879    5875 scope.go:122] "RemoveContainer" containerID="bdf942a19a3ec22f63d9946309f8408715eddf000e7ad2a71887f0f2ee1f65b5"
Nov 02 02:19:39.231043 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:19:39.231020    5875 scope.go:122] "RemoveContainer" containerID="d4243afb4ea6465f087a7f557ccd070c080324601fd74ce394da4c987dcb7835"
Nov 02 02:19:39.239281 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:19:39.239258    5875 scope.go:122] "RemoveContainer" containerID="f714ac9c3b89517edd66384dfe16095fd6bd550983e1c49ba6748d8976ea3cbe"
Nov 02 02:19:39.247067 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:19:39.247050    5875 scope.go:122] "RemoveContainer" containerID="f9d08cdbac86114fae4fbe47da664233f32ff88339710981c7402b57e3ff5b94"
Nov 02 02:19:39.255212 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:19:39.255190    5875 scope.go:122] "RemoveContainer" containerID="2f9d7c15014132ae1bb0e5ebba8efcace8e7f23da5ef64833263fbfe3f34f931"
Nov 02 02:19:41.318048 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:19:41.318011    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:19:44.013096 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:19:44.013062    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:19:54.999105 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:19:54.999068    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:20:01.047968 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:20:01.047924    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c96f480-h5lwz"]
Nov 02 02:20:01.096946 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:20:01.096916    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c96f480-h5lwz"]
Nov 02 02:20:02.694649 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:20:02.694615    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49894b1d-dea5-4978-b51f-b3e6f80b1326" path="/var/lib/kubelet/pods/49894b1d-dea5-4978-b51f-b3e6f80b1326/volumes"
Nov 02 02:20:11.317723 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:20:11.317683    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:20:15.496156 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:20:15.496118    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:20:30.047451 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:20:30.047411    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1a38c0d-2bpr7"]
Nov 02 02:20:30.096684 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:20:30.096648    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1a38c0d-2bpr7"]
Nov 02 02:20:30.693118 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:20:30.693088    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e9e1f5e-d36d-4e4e-a990-f9da53c59187" path="/var/lib/kubelet/pods/7e9e1f5e-d36d-4e4e-a990-f9da53c59187/volumes"
Nov 02 02:20:32.399995 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:20:32.399961    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:20:35.180385 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:20:35.180342    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ghslr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:20:35.181069 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:20:35.181033    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr" podUID="0c2d802d-8fbd-430d-ae72-29b0e217714d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:20:35.426247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:20:35.425350    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-727fm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:20:35.426247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:20:35.425452    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-727fm" podUID="a50af6e0-a390-47d4-a4d2-7f90a3055661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:20:35.531247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:20:35.530356    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-h8px6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:20:35.531247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:20:35.530460    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-h8px6" podUID="ba032348-8064-40c7-8278-0115c943882e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:20:39.290955 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:20:39.290925    5875 scope.go:122] "RemoveContainer" containerID="7bac62164b8601941d3d964cb0a86a6876a06178646a5f0296c1bed18baf3127"
Nov 02 02:20:41.317868 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:20:41.317777    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:20:45.046753 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:20:45.046719    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87e1e297-6t6hr"]
Nov 02 02:20:45.095999 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:20:45.095972    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87e1e297-6t6hr"]
Nov 02 02:20:46.693769 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:20:46.693732    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b9d0472-466e-444a-b2ee-309738376172" path="/var/lib/kubelet/pods/0b9d0472-466e-444a-b2ee-309738376172/volumes"
Nov 02 02:21:03.584186 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:21:03.584150    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:21:11.318463 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:21:11.318419    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:21:14.206163 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:21:14.206125    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:21:26.912914 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:21:26.912878    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:21:39.316321 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:21:39.316288    5875 scope.go:122] "RemoveContainer" containerID="41285e3c052d4b9de3d2d6837b2891816efffe3492561fbfab2677fcd3f5e826"
Nov 02 02:21:39.325103 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:21:39.325084    5875 scope.go:122] "RemoveContainer" containerID="b5d0849624a9ff1cf546353c6a690126d4f54e4c837cb938387a72be4c83ef27"
Nov 02 02:21:41.317480 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:21:41.317423    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:21:52.273027 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:21:52.272986    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:21:59.625434 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:21:59.625373    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-mfmmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:21:59.626176 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:21:59.626137    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj" podUID="004e401a-4d50-46ec-b1bd-d9896f9b20d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:22:02.103599 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:22:02.103568    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:22:11.318035 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:22:11.317944    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:22:23.402339 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:22:23.402307    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:22:31.050343 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:22:31.050299    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db1f4f70-2qx5t"]
Nov 02 02:22:31.098600 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:22:31.098563    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db1f4f70-2qx5t"]
Nov 02 02:22:32.705435 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:22:32.705398    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="857ac035-c585-49ba-aec3-d6d9987b1d0e" path="/var/lib/kubelet/pods/857ac035-c585-49ba-aec3-d6d9987b1d0e/volumes"
Nov 02 02:22:36.282190 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:22:36.282154    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:22:39.351297 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:22:39.351265    5875 scope.go:122] "RemoveContainer" containerID="5a7728e7a2216aef0f54704f467b702381ecd4c5b5944b21c03250f8905b84c4"
Nov 02 02:22:39.360109 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:22:39.360090    5875 scope.go:122] "RemoveContainer" containerID="5b03a84ae70dee47db76efef50cfbbf2e5b826e8ad5c3f486125f2d533cdbf74"
Nov 02 02:22:41.318056 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:22:41.318021    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:22:55.899650 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:22:55.899619    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:23:11.318882 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:23:11.318833    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:23:17.965904 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:23:17.965869    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:23:20.180370 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:23:20.180335    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ghslr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:23:20.180994 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:23:20.180961    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr" podUID="0c2d802d-8fbd-430d-ae72-29b0e217714d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:23:30.999966 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:23:30.999930    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:23:39.393792 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:23:39.393757    5875 scope.go:122] "RemoveContainer" containerID="238743f0dc145478d7262b2b9c7c63a7661ce3ce55aef562f42aeeb4d61f62a8"
Nov 02 02:23:39.445257 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:23:39.442062    5875 scope.go:122] "RemoveContainer" containerID="4f421defb1f349a3772a5f2325269f30019583d1d2e2357f47fa27be51d1bb56"
Nov 02 02:23:39.467135 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:23:39.467044    5875 scope.go:122] "RemoveContainer" containerID="ab50c6af463e0296d3bef9735b3de9254a9994ab16c8fd8eeb05c6fa51296ab8"
Nov 02 02:23:41.321733 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:23:41.321473    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:23:42.504148 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:23:42.504117    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:24:04.902096 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:24:04.902059    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:24:11.317739 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:24:11.317699    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:24:15.194025 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:24:15.193990    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:24:31.761194 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:24:31.761153    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:24:41.317336 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:24:41.317297    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:24:53.311440 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:24:53.311345    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:25:03.979969 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:03.979932    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:25:11.318401 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:25:11.318320    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:25:22.892686 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:22.892645    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:25:33.598280 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:33.598242    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:25:40.996515 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:40.996478    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"]
Nov 02 02:25:41.000329 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.000307    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.055512 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.055479    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-054a89-openshift-default-dockercfg-wnxjt\""
Nov 02 02:25:41.104403 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.104371    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"]
Nov 02 02:25:41.176921 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.176884    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-workload-socket\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.177105 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.176929    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-data\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.177105 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.177034    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-envoy\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.177105 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.177073    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-credential-socket\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.177105 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.177096    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-token\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.177328 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.177117    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-podinfo\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.177328 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.177296    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-workload-certs\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.177328 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.177321    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313d2d27-4421-4dbf-bce7-f5186d12c073-istiod-ca-cert\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.177505 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.177353    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65zfl\" (UniqueName: \"kubernetes.io/projected/313d2d27-4421-4dbf-bce7-f5186d12c073-kube-api-access-65zfl\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.278122 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.278043    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65zfl\" (UniqueName: \"kubernetes.io/projected/313d2d27-4421-4dbf-bce7-f5186d12c073-kube-api-access-65zfl\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.278122 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.278085    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-workload-socket\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.278122 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.278120    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-data\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.278434 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.278185    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-envoy\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.278434 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.278260    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-credential-socket\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.278434 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.278304    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-token\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.278434 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.278336    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-podinfo\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.278434 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.278413    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-workload-certs\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.278672 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.278451    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313d2d27-4421-4dbf-bce7-f5186d12c073-istiod-ca-cert\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.278672 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.278549    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-workload-socket\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.278796 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.278770    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-credential-socket\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.279030 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.279004    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-data\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.279148 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.279062    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-workload-certs\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.279260 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.279209    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313d2d27-4421-4dbf-bce7-f5186d12c073-istiod-ca-cert\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.281070 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.281041    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-envoy\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.281255 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.281151    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-podinfo\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.319376 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:25:41.319341    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:25:41.339990 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.339959    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-token\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.340192 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.339992    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65zfl\" (UniqueName: \"kubernetes.io/projected/313d2d27-4421-4dbf-bce7-f5186d12c073-kube-api-access-65zfl\") pod \"e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") " pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:41.613862 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:41.613774    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:42.045868 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:42.045844    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"]
Nov 02 02:25:42.048070 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:25:42.048041    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313d2d27_4421_4dbf_bce7_f5186d12c073.slice/crio-71cde57ea679a4cc0d54dafb0cc09263400766db668bdc0c00c0ac82f4b14a04 WatchSource:0}: Error finding container 71cde57ea679a4cc0d54dafb0cc09263400766db668bdc0c00c0ac82f4b14a04: Status 404 returned error can't find the container with id 71cde57ea679a4cc0d54dafb0cc09263400766db668bdc0c00c0ac82f4b14a04
Nov 02 02:25:42.050313 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:42.050281    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:25:42.050388 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:42.050349    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:25:42.050388 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:42.050379    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:25:42.282085 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:42.282045    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f" event={"ID":"313d2d27-4421-4dbf-bce7-f5186d12c073","Type":"ContainerStarted","Data":"70f98cc2862ccf995c313f9b603eea2eb6a04d00b6b9279b93982b8e1d3be817"}
Nov 02 02:25:42.282267 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:42.282113    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f" event={"ID":"313d2d27-4421-4dbf-bce7-f5186d12c073","Type":"ContainerStarted","Data":"71cde57ea679a4cc0d54dafb0cc09263400766db668bdc0c00c0ac82f4b14a04"}
Nov 02 02:25:42.403647 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:42.403551    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f" podStartSLOduration=2.403532227 podStartE2EDuration="2.403532227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:25:42.40188069 +0000 UTC m=+3189.842917201" watchObservedRunningTime="2026-11-02 02:25:42.403532227 +0000 UTC m=+3189.844568720"
Nov 02 02:25:42.614777 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:42.614740    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:43.618964 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:43.618932    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:44.292278 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:44.292250    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:44.293962 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:44.293940    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:25:59.762018 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:25:59.761976    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:26:10.444186 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:10.444150    5875 generic.go:356] "Generic (PLEG): container finished" podID="313d2d27-4421-4dbf-bce7-f5186d12c073" containerID="70f98cc2862ccf995c313f9b603eea2eb6a04d00b6b9279b93982b8e1d3be817" exitCode=0
Nov 02 02:26:10.444712 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:10.444206    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f" event={"ID":"313d2d27-4421-4dbf-bce7-f5186d12c073","Type":"ContainerDied","Data":"70f98cc2862ccf995c313f9b603eea2eb6a04d00b6b9279b93982b8e1d3be817"}
Nov 02 02:26:10.444712 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:10.444618    5875 scope.go:122] "RemoveContainer" containerID="70f98cc2862ccf995c313f9b603eea2eb6a04d00b6b9279b93982b8e1d3be817"
Nov 02 02:26:10.445886 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:10.445848    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:26:10.445966 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:10.445928    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:26:10.445966 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:10.445961    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:26:11.346467 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:26:11.346422    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:26:11.451365 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:11.451331    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f" event={"ID":"313d2d27-4421-4dbf-bce7-f5186d12c073","Type":"ContainerStarted","Data":"80cf0c7194a3701c2ffb9e89e18e9ea2eb8755cfaa2c978bd16175e0aaf89c22"}
Nov 02 02:26:11.622307 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:11.619401    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:26:11.622508 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:11.622388    5875 patch_prober.go:29] interesting pod/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 02 02:26:11.622508 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:11.622469    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f" podUID="313d2d27-4421-4dbf-bce7-f5186d12c073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 02 02:26:12.623366 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:12.623331    5875 patch_prober.go:29] interesting pod/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 02 02:26:12.623789 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:12.623390    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f" podUID="313d2d27-4421-4dbf-bce7-f5186d12c073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 02 02:26:13.621457 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:13.621418    5875 patch_prober.go:29] interesting pod/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 02 02:26:13.621751 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:13.621724    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f" podUID="313d2d27-4421-4dbf-bce7-f5186d12c073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 02 02:26:14.292680 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:14.292647    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:26:14.619303 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:14.615632    5875 patch_prober.go:29] interesting pod/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 02 02:26:14.619303 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:14.615693    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f" podUID="313d2d27-4421-4dbf-bce7-f5186d12c073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 02 02:26:15.618245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:15.617602    5875 patch_prober.go:29] interesting pod/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 02 02:26:15.618245 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:15.617664    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f" podUID="313d2d27-4421-4dbf-bce7-f5186d12c073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 02 02:26:16.617969 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:16.617937    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:26:16.618982 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:16.618958    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:26:18.065513 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:18.065481    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:26:41.318359 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:26:41.318270    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:26:44.228724 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:44.228679    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:26:58.287644 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:58.287604    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3b32f86-h6swk"]
Nov 02 02:26:58.291443 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:58.291426    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b32f86-h6swk"
Nov 02 02:26:58.343492 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:58.343463    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:26:58.369314 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:58.369284    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3b32f86-h6swk\" (UID: \"22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b32f86-h6swk"
Nov 02 02:26:58.369447 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:58.369361    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3b32f86-h6swk\" (UID: \"22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b32f86-h6swk"
Nov 02 02:26:58.369447 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:58.369422    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwhm6\" (UniqueName: \"kubernetes.io/projected/22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569-kube-api-access-kwhm6\") pod \"maas-api-revoke-sub-a3b32f86-h6swk\" (UID: \"22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b32f86-h6swk"
Nov 02 02:26:58.392973 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:58.392932    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:26:58.392973 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:58.392932    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:26:58.393186 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:58.393004    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:26:58.394848 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:58.394822    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3b32f86-h6swk"]
Nov 02 02:26:58.469930 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:58.469905    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3b32f86-h6swk\" (UID: \"22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b32f86-h6swk"
Nov 02 02:26:58.470051 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:58.469944    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwhm6\" (UniqueName: \"kubernetes.io/projected/22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569-kube-api-access-kwhm6\") pod \"maas-api-revoke-sub-a3b32f86-h6swk\" (UID: \"22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b32f86-h6swk"
Nov 02 02:26:58.470051 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:58.470029    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3b32f86-h6swk\" (UID: \"22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b32f86-h6swk"
Nov 02 02:26:58.470741 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:58.470685    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3b32f86-h6swk\" (UID: \"22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b32f86-h6swk"
Nov 02 02:26:58.472312 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:58.472293    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3b32f86-h6swk\" (UID: \"22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b32f86-h6swk"
Nov 02 02:26:58.531813 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:58.531783    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwhm6\" (UniqueName: \"kubernetes.io/projected/22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569-kube-api-access-kwhm6\") pod \"maas-api-revoke-sub-a3b32f86-h6swk\" (UID: \"22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b32f86-h6swk"
Nov 02 02:26:58.605058 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:58.604997    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b32f86-h6swk"
Nov 02 02:26:58.977996 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:58.977971    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3b32f86-h6swk"]
Nov 02 02:26:58.980362 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:26:58.980330    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b39ba1_7a7d_4fb1_bb18_b0bdb36b8569.slice/crio-388f6748ea90bb5c524ae2527bae11bb66663ead3134396c25b59d5124f64aa3 WatchSource:0}: Error finding container 388f6748ea90bb5c524ae2527bae11bb66663ead3134396c25b59d5124f64aa3: Status 404 returned error can't find the container with id 388f6748ea90bb5c524ae2527bae11bb66663ead3134396c25b59d5124f64aa3
Nov 02 02:26:59.700479 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:59.700380    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b32f86-h6swk" event={"ID":"22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569","Type":"ContainerStarted","Data":"f16db435dd06551afd9afd70dc9d43d8bd57bfd4e23fbc6ce5ee90f3b196bf0d"}
Nov 02 02:26:59.700479 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:26:59.700464    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b32f86-h6swk" event={"ID":"22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569","Type":"ContainerStarted","Data":"388f6748ea90bb5c524ae2527bae11bb66663ead3134396c25b59d5124f64aa3"}
Nov 02 02:27:00.706869 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:00.706836    5875 generic.go:356] "Generic (PLEG): container finished" podID="22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569" containerID="f16db435dd06551afd9afd70dc9d43d8bd57bfd4e23fbc6ce5ee90f3b196bf0d" exitCode=0
Nov 02 02:27:00.707334 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:00.706913    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b32f86-h6swk" event={"ID":"22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569","Type":"ContainerDied","Data":"f16db435dd06551afd9afd70dc9d43d8bd57bfd4e23fbc6ce5ee90f3b196bf0d"}
Nov 02 02:27:01.851881 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:01.851857    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b32f86-h6swk"
Nov 02 02:27:02.006621 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:02.006542    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569-maas-api-xks-ca\") pod \"22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569\" (UID: \"22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569\") "
Nov 02 02:27:02.006776 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:02.006642    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569-kube-api-access-kwhm6\" (UniqueName: \"kubernetes.io/projected/22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569-kube-api-access-kwhm6\") pod \"22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569\" (UID: \"22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569\") "
Nov 02 02:27:02.006824 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:02.006773    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569-maas-api-openshift-ca\") pod \"22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569\" (UID: \"22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569\") "
Nov 02 02:27:02.007102 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:02.007075    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569-maas-api-openshift-ca" pod "22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569" (UID: "22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:27:02.008787 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:02.008763    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569-maas-api-xks-ca" pod "22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569" (UID: "22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:27:02.008873 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:02.008832    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569-kube-api-access-kwhm6" pod "22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569" (UID: "22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569"). InnerVolumeSpecName "kube-api-access-kwhm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:27:02.108032 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:02.108006    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:02.108032 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:02.108031    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwhm6\" (UniqueName: \"kubernetes.io/projected/22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569-kube-api-access-kwhm6\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:02.108276 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:02.108044    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:02.720414 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:02.720387    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b32f86-h6swk"
Nov 02 02:27:02.720414 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:02.720403    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b32f86-h6swk" event={"ID":"22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569","Type":"ContainerDied","Data":"388f6748ea90bb5c524ae2527bae11bb66663ead3134396c25b59d5124f64aa3"}
Nov 02 02:27:02.720622 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:02.720444    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="388f6748ea90bb5c524ae2527bae11bb66663ead3134396c25b59d5124f64aa3"
Nov 02 02:27:10.491136 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:10.491102    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr"]
Nov 02 02:27:10.491632 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:10.491618    5875 cpu_manager.go:414] "containerMap: removing container" podUID="22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569" containerName="revoke-keys"
Nov 02 02:27:10.491685 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:10.491633    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569" containerName="revoke-keys"
Nov 02 02:27:10.491725 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:10.491703    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569" containerName="revoke-keys"
Nov 02 02:27:10.494798 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:10.494783    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr"
Nov 02 02:27:10.500588 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:10.500560    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmfzt\" (UniqueName: \"kubernetes.io/projected/a944eaba-fc2b-4dd8-9dba-d8566de7c683-kube-api-access-lmfzt\") pod \"maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr\" (UID: \"a944eaba-fc2b-4dd8-9dba-d8566de7c683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr"
Nov 02 02:27:10.500726 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:10.500611    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a944eaba-fc2b-4dd8-9dba-d8566de7c683-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr\" (UID: \"a944eaba-fc2b-4dd8-9dba-d8566de7c683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr"
Nov 02 02:27:10.500726 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:10.500691    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a944eaba-fc2b-4dd8-9dba-d8566de7c683-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr\" (UID: \"a944eaba-fc2b-4dd8-9dba-d8566de7c683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr"
Nov 02 02:27:10.546638 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:10.546612    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:27:10.596192 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:10.596161    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:27:10.596192 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:10.596190    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:27:10.596422 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:10.596253    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:27:10.598050 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:10.598028    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr"]
Nov 02 02:27:10.601938 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:10.601919    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmfzt\" (UniqueName: \"kubernetes.io/projected/a944eaba-fc2b-4dd8-9dba-d8566de7c683-kube-api-access-lmfzt\") pod \"maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr\" (UID: \"a944eaba-fc2b-4dd8-9dba-d8566de7c683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr"
Nov 02 02:27:10.602006 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:10.601958    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a944eaba-fc2b-4dd8-9dba-d8566de7c683-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr\" (UID: \"a944eaba-fc2b-4dd8-9dba-d8566de7c683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr"
Nov 02 02:27:10.602111 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:10.602094    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a944eaba-fc2b-4dd8-9dba-d8566de7c683-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr\" (UID: \"a944eaba-fc2b-4dd8-9dba-d8566de7c683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr"
Nov 02 02:27:10.602603 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:10.602585    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a944eaba-fc2b-4dd8-9dba-d8566de7c683-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr\" (UID: \"a944eaba-fc2b-4dd8-9dba-d8566de7c683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr"
Nov 02 02:27:10.604652 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:10.604628    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a944eaba-fc2b-4dd8-9dba-d8566de7c683-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr\" (UID: \"a944eaba-fc2b-4dd8-9dba-d8566de7c683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr"
Nov 02 02:27:10.665182 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:10.665152    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmfzt\" (UniqueName: \"kubernetes.io/projected/a944eaba-fc2b-4dd8-9dba-d8566de7c683-kube-api-access-lmfzt\") pod \"maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr\" (UID: \"a944eaba-fc2b-4dd8-9dba-d8566de7c683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr"
Nov 02 02:27:10.807954 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:10.807886    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr"
Nov 02 02:27:11.186479 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:11.186426    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr"]
Nov 02 02:27:11.188916 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:27:11.188891    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda944eaba_fc2b_4dd8_9dba_d8566de7c683.slice/crio-6049ca73f8bc857e885695267a687a5cfa83604cf659eccc81218d691dd32b8e WatchSource:0}: Error finding container 6049ca73f8bc857e885695267a687a5cfa83604cf659eccc81218d691dd32b8e: Status 404 returned error can't find the container with id 6049ca73f8bc857e885695267a687a5cfa83604cf659eccc81218d691dd32b8e
Nov 02 02:27:11.318803 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:27:11.318764    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:27:11.765123 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:11.765039    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr" event={"ID":"a944eaba-fc2b-4dd8-9dba-d8566de7c683","Type":"ContainerStarted","Data":"55a9529d636fc271a871732aed393fadc2227eb2899c60bb3f8a7e8f527450c7"}
Nov 02 02:27:11.765123 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:11.765088    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr" event={"ID":"a944eaba-fc2b-4dd8-9dba-d8566de7c683","Type":"ContainerStarted","Data":"6049ca73f8bc857e885695267a687a5cfa83604cf659eccc81218d691dd32b8e"}
Nov 02 02:27:11.880732 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:11.880675    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr" podStartSLOduration=1.88065971 podStartE2EDuration="1.88065971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:27:11.879563182 +0000 UTC m=+3279.320599680" watchObservedRunningTime="2026-11-02 02:27:11.88065971 +0000 UTC m=+3279.321696204"
Nov 02 02:27:12.771712 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:12.771680    5875 generic.go:356] "Generic (PLEG): container finished" podID="a944eaba-fc2b-4dd8-9dba-d8566de7c683" containerID="55a9529d636fc271a871732aed393fadc2227eb2899c60bb3f8a7e8f527450c7" exitCode=0
Nov 02 02:27:12.772144 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:12.771743    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr" event={"ID":"a944eaba-fc2b-4dd8-9dba-d8566de7c683","Type":"ContainerDied","Data":"55a9529d636fc271a871732aed393fadc2227eb2899c60bb3f8a7e8f527450c7"}
Nov 02 02:27:13.330424 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:13.329733    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:27:13.923974 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:13.923950    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr"
Nov 02 02:27:13.940106 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:13.940081    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a944eaba-fc2b-4dd8-9dba-d8566de7c683-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a944eaba-fc2b-4dd8-9dba-d8566de7c683-maas-api-xks-ca\") pod \"a944eaba-fc2b-4dd8-9dba-d8566de7c683\" (UID: \"a944eaba-fc2b-4dd8-9dba-d8566de7c683\") "
Nov 02 02:27:13.940275 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:13.940242    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a944eaba-fc2b-4dd8-9dba-d8566de7c683-kube-api-access-lmfzt\" (UniqueName: \"kubernetes.io/projected/a944eaba-fc2b-4dd8-9dba-d8566de7c683-kube-api-access-lmfzt\") pod \"a944eaba-fc2b-4dd8-9dba-d8566de7c683\" (UID: \"a944eaba-fc2b-4dd8-9dba-d8566de7c683\") "
Nov 02 02:27:13.940350 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:13.940300    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a944eaba-fc2b-4dd8-9dba-d8566de7c683-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a944eaba-fc2b-4dd8-9dba-d8566de7c683-maas-api-openshift-ca\") pod \"a944eaba-fc2b-4dd8-9dba-d8566de7c683\" (UID: \"a944eaba-fc2b-4dd8-9dba-d8566de7c683\") "
Nov 02 02:27:13.940624 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:13.940602    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a944eaba-fc2b-4dd8-9dba-d8566de7c683-maas-api-openshift-ca" pod "a944eaba-fc2b-4dd8-9dba-d8566de7c683" (UID: "a944eaba-fc2b-4dd8-9dba-d8566de7c683"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:27:13.942162 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:13.942144    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a944eaba-fc2b-4dd8-9dba-d8566de7c683-maas-api-xks-ca" pod "a944eaba-fc2b-4dd8-9dba-d8566de7c683" (UID: "a944eaba-fc2b-4dd8-9dba-d8566de7c683"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:27:13.942364 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:13.942342    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a944eaba-fc2b-4dd8-9dba-d8566de7c683-kube-api-access-lmfzt" pod "a944eaba-fc2b-4dd8-9dba-d8566de7c683" (UID: "a944eaba-fc2b-4dd8-9dba-d8566de7c683"). InnerVolumeSpecName "kube-api-access-lmfzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:27:14.041722 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:14.041665    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a944eaba-fc2b-4dd8-9dba-d8566de7c683-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:14.041722 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:14.041688    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmfzt\" (UniqueName: \"kubernetes.io/projected/a944eaba-fc2b-4dd8-9dba-d8566de7c683-kube-api-access-lmfzt\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:14.041722 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:14.041699    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a944eaba-fc2b-4dd8-9dba-d8566de7c683-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:14.783859 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:14.783826    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr"
Nov 02 02:27:14.784041 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:14.783837    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr" event={"ID":"a944eaba-fc2b-4dd8-9dba-d8566de7c683","Type":"ContainerDied","Data":"6049ca73f8bc857e885695267a687a5cfa83604cf659eccc81218d691dd32b8e"}
Nov 02 02:27:14.784041 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:14.783948    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6049ca73f8bc857e885695267a687a5cfa83604cf659eccc81218d691dd32b8e"
Nov 02 02:27:30.904514 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:30.904472    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr"]
Nov 02 02:27:30.944855 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:30.944827    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-054a89-lx2kr"]
Nov 02 02:27:32.693440 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:32.693406    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a944eaba-fc2b-4dd8-9dba-d8566de7c683" path="/var/lib/kubelet/pods/a944eaba-fc2b-4dd8-9dba-d8566de7c683/volumes"
Nov 02 02:27:33.404166 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:33.404127    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:27:37.541287 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:37.541252    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"]
Nov 02 02:27:37.541724 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:37.541513    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f" podUID="313d2d27-4421-4dbf-bce7-f5186d12c073" containerName="istio-proxy" containerID="cri-o://80cf0c7194a3701c2ffb9e89e18e9ea2eb8755cfaa2c978bd16175e0aaf89c22" gracePeriod=30
Nov 02 02:27:41.317436 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:27:41.317393    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:27:42.800341 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.800319    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:27:42.881540 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.881454    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/313d2d27-4421-4dbf-bce7-f5186d12c073-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313d2d27-4421-4dbf-bce7-f5186d12c073-istiod-ca-cert\") pod \"313d2d27-4421-4dbf-bce7-f5186d12c073\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") "
Nov 02 02:27:42.881540 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.881515    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-data\") pod \"313d2d27-4421-4dbf-bce7-f5186d12c073\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") "
Nov 02 02:27:42.881750 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.881563    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-credential-socket\") pod \"313d2d27-4421-4dbf-bce7-f5186d12c073\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") "
Nov 02 02:27:42.881750 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.881589    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-workload-certs\") pod \"313d2d27-4421-4dbf-bce7-f5186d12c073\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") "
Nov 02 02:27:42.881750 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.881702    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-workload-socket\") pod \"313d2d27-4421-4dbf-bce7-f5186d12c073\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") "
Nov 02 02:27:42.881891 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.881749    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-envoy\") pod \"313d2d27-4421-4dbf-bce7-f5186d12c073\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") "
Nov 02 02:27:42.881891 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.881781    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/313d2d27-4421-4dbf-bce7-f5186d12c073-kube-api-access-65zfl\" (UniqueName: \"kubernetes.io/projected/313d2d27-4421-4dbf-bce7-f5186d12c073-kube-api-access-65zfl\") pod \"313d2d27-4421-4dbf-bce7-f5186d12c073\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") "
Nov 02 02:27:42.881891 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.881828    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-token\" (UniqueName: \"kubernetes.io/projected/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-token\") pod \"313d2d27-4421-4dbf-bce7-f5186d12c073\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") "
Nov 02 02:27:42.881891 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.881829    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-data" pod "313d2d27-4421-4dbf-bce7-f5186d12c073" (UID: "313d2d27-4421-4dbf-bce7-f5186d12c073"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:27:42.882091 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.881899    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-credential-socket" pod "313d2d27-4421-4dbf-bce7-f5186d12c073" (UID: "313d2d27-4421-4dbf-bce7-f5186d12c073"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:27:42.882091 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.881931    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-podinfo\") pod \"313d2d27-4421-4dbf-bce7-f5186d12c073\" (UID: \"313d2d27-4421-4dbf-bce7-f5186d12c073\") "
Nov 02 02:27:42.882091 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.881950    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/313d2d27-4421-4dbf-bce7-f5186d12c073-istiod-ca-cert" pod "313d2d27-4421-4dbf-bce7-f5186d12c073" (UID: "313d2d27-4421-4dbf-bce7-f5186d12c073"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:27:42.882091 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.881924    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-workload-certs" pod "313d2d27-4421-4dbf-bce7-f5186d12c073" (UID: "313d2d27-4421-4dbf-bce7-f5186d12c073"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:27:42.882091 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.881973    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-workload-socket" pod "313d2d27-4421-4dbf-bce7-f5186d12c073" (UID: "313d2d27-4421-4dbf-bce7-f5186d12c073"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:27:42.882397 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.882365    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:42.882397 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.882382    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313d2d27-4421-4dbf-bce7-f5186d12c073-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:42.882397 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.882391    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:42.882737 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.882404    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:42.882737 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.882416    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:42.884125 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.884098    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-envoy" pod "313d2d27-4421-4dbf-bce7-f5186d12c073" (UID: "313d2d27-4421-4dbf-bce7-f5186d12c073"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:27:42.884318 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.884196    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-token" pod "313d2d27-4421-4dbf-bce7-f5186d12c073" (UID: "313d2d27-4421-4dbf-bce7-f5186d12c073"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:27:42.884573 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.884557    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-podinfo" pod "313d2d27-4421-4dbf-bce7-f5186d12c073" (UID: "313d2d27-4421-4dbf-bce7-f5186d12c073"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:27:42.884644 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.884592    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313d2d27-4421-4dbf-bce7-f5186d12c073-kube-api-access-65zfl" pod "313d2d27-4421-4dbf-bce7-f5186d12c073" (UID: "313d2d27-4421-4dbf-bce7-f5186d12c073"). InnerVolumeSpecName "kube-api-access-65zfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:27:42.952409 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.952375    5875 generic.go:356] "Generic (PLEG): container finished" podID="313d2d27-4421-4dbf-bce7-f5186d12c073" containerID="80cf0c7194a3701c2ffb9e89e18e9ea2eb8755cfaa2c978bd16175e0aaf89c22" exitCode=0
Nov 02 02:27:42.952625 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.952433    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"
Nov 02 02:27:42.952625 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.952459    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f" event={"ID":"313d2d27-4421-4dbf-bce7-f5186d12c073","Type":"ContainerDied","Data":"80cf0c7194a3701c2ffb9e89e18e9ea2eb8755cfaa2c978bd16175e0aaf89c22"}
Nov 02 02:27:42.952625 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.952510    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f" event={"ID":"313d2d27-4421-4dbf-bce7-f5186d12c073","Type":"ContainerDied","Data":"71cde57ea679a4cc0d54dafb0cc09263400766db668bdc0c00c0ac82f4b14a04"}
Nov 02 02:27:42.952625 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.952533    5875 scope.go:122] "RemoveContainer" containerID="80cf0c7194a3701c2ffb9e89e18e9ea2eb8755cfaa2c978bd16175e0aaf89c22"
Nov 02 02:27:42.962360 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.962341    5875 scope.go:122] "RemoveContainer" containerID="70f98cc2862ccf995c313f9b603eea2eb6a04d00b6b9279b93982b8e1d3be817"
Nov 02 02:27:42.972256 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.971538    5875 scope.go:122] "RemoveContainer" containerID="80cf0c7194a3701c2ffb9e89e18e9ea2eb8755cfaa2c978bd16175e0aaf89c22"
Nov 02 02:27:42.972256 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:27:42.971864    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80cf0c7194a3701c2ffb9e89e18e9ea2eb8755cfaa2c978bd16175e0aaf89c22\": container with ID starting with 80cf0c7194a3701c2ffb9e89e18e9ea2eb8755cfaa2c978bd16175e0aaf89c22 not found: ID does not exist" containerID="80cf0c7194a3701c2ffb9e89e18e9ea2eb8755cfaa2c978bd16175e0aaf89c22"
Nov 02 02:27:42.972256 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.971897    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80cf0c7194a3701c2ffb9e89e18e9ea2eb8755cfaa2c978bd16175e0aaf89c22"} err="failed to get container status \"80cf0c7194a3701c2ffb9e89e18e9ea2eb8755cfaa2c978bd16175e0aaf89c22\": rpc error: code = NotFound desc = could not find container \"80cf0c7194a3701c2ffb9e89e18e9ea2eb8755cfaa2c978bd16175e0aaf89c22\": container with ID starting with 80cf0c7194a3701c2ffb9e89e18e9ea2eb8755cfaa2c978bd16175e0aaf89c22 not found: ID does not exist"
Nov 02 02:27:42.972256 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.971923    5875 scope.go:122] "RemoveContainer" containerID="70f98cc2862ccf995c313f9b603eea2eb6a04d00b6b9279b93982b8e1d3be817"
Nov 02 02:27:42.972632 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:27:42.972600    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70f98cc2862ccf995c313f9b603eea2eb6a04d00b6b9279b93982b8e1d3be817\": container with ID starting with 70f98cc2862ccf995c313f9b603eea2eb6a04d00b6b9279b93982b8e1d3be817 not found: ID does not exist" containerID="70f98cc2862ccf995c313f9b603eea2eb6a04d00b6b9279b93982b8e1d3be817"
Nov 02 02:27:42.972773 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.972638    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f98cc2862ccf995c313f9b603eea2eb6a04d00b6b9279b93982b8e1d3be817"} err="failed to get container status \"70f98cc2862ccf995c313f9b603eea2eb6a04d00b6b9279b93982b8e1d3be817\": rpc error: code = NotFound desc = could not find container \"70f98cc2862ccf995c313f9b603eea2eb6a04d00b6b9279b93982b8e1d3be817\": container with ID starting with 70f98cc2862ccf995c313f9b603eea2eb6a04d00b6b9279b93982b8e1d3be817 not found: ID does not exist"
Nov 02 02:27:42.983375 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.983351    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:42.983515 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.983378    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65zfl\" (UniqueName: \"kubernetes.io/projected/313d2d27-4421-4dbf-bce7-f5186d12c073-kube-api-access-65zfl\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:42.983515 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.983392    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:42.983515 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:42.983404    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313d2d27-4421-4dbf-bce7-f5186d12c073-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:27:43.084475 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:43.084434    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"]
Nov 02 02:27:43.091413 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:43.091387    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-054a89-openshift-default-576b9d889cx9k9f"]
Nov 02 02:27:44.692595 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:27:44.692553    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="313d2d27-4421-4dbf-bce7-f5186d12c073" path="/var/lib/kubelet/pods/313d2d27-4421-4dbf-bce7-f5186d12c073/volumes"
Nov 02 02:28:11.317933 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:28:11.317850    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:28:41.317239 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:28:41.317172    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:29:11.318118 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:29:11.318069    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:29:27.207607 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:29:27.207569    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:29:29.627401 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:29:29.627355    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-mfmmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:29:29.628112 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:29:29.628074    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj" podUID="004e401a-4d50-46ec-b1bd-d9896f9b20d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:29:41.318042 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:29:41.317958    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:29:44.770912 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:29:44.770859    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:29:54.279534 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:29:54.279497    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:30:00.934256 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:00.934199    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-72fr5"]
Nov 02 02:30:00.934896 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:00.934864    5875 cpu_manager.go:414] "containerMap: removing container" podUID="313d2d27-4421-4dbf-bce7-f5186d12c073" containerName="istio-proxy"
Nov 02 02:30:00.934988 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:00.934898    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="313d2d27-4421-4dbf-bce7-f5186d12c073" containerName="istio-proxy"
Nov 02 02:30:00.934988 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:00.934922    5875 cpu_manager.go:414] "containerMap: removing container" podUID="a944eaba-fc2b-4dd8-9dba-d8566de7c683" containerName="revoke-keys"
Nov 02 02:30:00.934988 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:00.934930    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a944eaba-fc2b-4dd8-9dba-d8566de7c683" containerName="revoke-keys"
Nov 02 02:30:00.934988 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:00.934967    5875 cpu_manager.go:414] "containerMap: removing container" podUID="313d2d27-4421-4dbf-bce7-f5186d12c073" containerName="istio-proxy"
Nov 02 02:30:00.934988 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:00.934975    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="313d2d27-4421-4dbf-bce7-f5186d12c073" containerName="istio-proxy"
Nov 02 02:30:00.935286 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:00.935103    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="313d2d27-4421-4dbf-bce7-f5186d12c073" containerName="istio-proxy"
Nov 02 02:30:00.935286 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:00.935117    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="a944eaba-fc2b-4dd8-9dba-d8566de7c683" containerName="revoke-keys"
Nov 02 02:30:00.935286 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:00.935130    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="313d2d27-4421-4dbf-bce7-f5186d12c073" containerName="istio-proxy"
Nov 02 02:30:00.938499 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:00.938480    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-72fr5"
Nov 02 02:30:00.990606 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:00.990572    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:30:01.040094 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:01.040062    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:30:01.041975 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:01.041952    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-72fr5"]
Nov 02 02:30:01.087464 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:01.087427    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-72fr5\" (UID: \"d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-72fr5"
Nov 02 02:30:01.087619 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:01.087474    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-72fr5\" (UID: \"d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-72fr5"
Nov 02 02:30:01.087619 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:01.087495    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdsvl\" (UniqueName: \"kubernetes.io/projected/d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0-kube-api-access-xdsvl\") pod \"maas-api-revoke-sub-d2bc93bc-72fr5\" (UID: \"d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-72fr5"
Nov 02 02:30:01.091525 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:01.091501    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:30:01.091575 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:01.091545    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:30:01.189079 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:01.188994    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-72fr5\" (UID: \"d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-72fr5"
Nov 02 02:30:01.189079 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:01.189045    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-72fr5\" (UID: \"d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-72fr5"
Nov 02 02:30:01.189317 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:01.189169    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdsvl\" (UniqueName: \"kubernetes.io/projected/d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0-kube-api-access-xdsvl\") pod \"maas-api-revoke-sub-d2bc93bc-72fr5\" (UID: \"d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-72fr5"
Nov 02 02:30:01.189677 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:01.189657    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-72fr5\" (UID: \"d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-72fr5"
Nov 02 02:30:01.191594 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:01.191577    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-72fr5\" (UID: \"d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-72fr5"
Nov 02 02:30:01.250177 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:01.250151    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdsvl\" (UniqueName: \"kubernetes.io/projected/d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0-kube-api-access-xdsvl\") pod \"maas-api-revoke-sub-d2bc93bc-72fr5\" (UID: \"d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-72fr5"
Nov 02 02:30:01.250978 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:01.250960    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-72fr5"
Nov 02 02:30:01.627586 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:01.627558    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-72fr5"]
Nov 02 02:30:01.628934 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:30:01.628906    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ebdd42_2ac4_4fc1_9934_445b4bfe12a0.slice/crio-a01a9307eee490a4995ef770399f5a96550301c37ac31e5a1865bd17f7cdae07 WatchSource:0}: Error finding container a01a9307eee490a4995ef770399f5a96550301c37ac31e5a1865bd17f7cdae07: Status 404 returned error can't find the container with id a01a9307eee490a4995ef770399f5a96550301c37ac31e5a1865bd17f7cdae07
Nov 02 02:30:01.687188 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:01.687160    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-72fr5" event={"ID":"d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0","Type":"ContainerStarted","Data":"a01a9307eee490a4995ef770399f5a96550301c37ac31e5a1865bd17f7cdae07"}
Nov 02 02:30:02.694089 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:02.694048    5875 generic.go:356] "Generic (PLEG): container finished" podID="d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0" containerID="d0e3cdfbb962e9b600ab5e01b459ab5eeff35b06b0c14a258f825c91d2207a66" exitCode=0
Nov 02 02:30:02.694795 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:02.694773    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-72fr5" event={"ID":"d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0","Type":"ContainerDied","Data":"d0e3cdfbb962e9b600ab5e01b459ab5eeff35b06b0c14a258f825c91d2207a66"}
Nov 02 02:30:03.840544 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:03.840517    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-72fr5"
Nov 02 02:30:03.916715 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:03.916679    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0-maas-api-openshift-ca\") pod \"d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0\" (UID: \"d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0\") "
Nov 02 02:30:03.916715 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:03.916719    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0-kube-api-access-xdsvl\" (UniqueName: \"kubernetes.io/projected/d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0-kube-api-access-xdsvl\") pod \"d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0\" (UID: \"d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0\") "
Nov 02 02:30:03.916946 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:03.916799    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0-maas-api-xks-ca\") pod \"d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0\" (UID: \"d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0\") "
Nov 02 02:30:03.917137 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:03.917108    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0-maas-api-openshift-ca" pod "d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0" (UID: "d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:30:03.919011 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:03.918957    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0-kube-api-access-xdsvl" pod "d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0" (UID: "d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0"). InnerVolumeSpecName "kube-api-access-xdsvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:30:03.919011 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:03.918997    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0-maas-api-xks-ca" pod "d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0" (UID: "d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:30:04.018486 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:04.018452    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:30:04.018486 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:04.018485    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdsvl\" (UniqueName: \"kubernetes.io/projected/d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0-kube-api-access-xdsvl\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:30:04.018714 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:04.018500    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:30:04.705045 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:04.705018    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-72fr5" event={"ID":"d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0","Type":"ContainerDied","Data":"a01a9307eee490a4995ef770399f5a96550301c37ac31e5a1865bd17f7cdae07"}
Nov 02 02:30:04.705201 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:04.705055    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-72fr5"
Nov 02 02:30:04.705201 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:04.705062    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a01a9307eee490a4995ef770399f5a96550301c37ac31e5a1865bd17f7cdae07"
Nov 02 02:30:11.318422 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:30:11.318381    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:30:13.765912 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:13.765876    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:30:14.049772 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.049694    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"]
Nov 02 02:30:14.050137 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.050123    5875 cpu_manager.go:414] "containerMap: removing container" podUID="d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0" containerName="revoke-keys"
Nov 02 02:30:14.050194 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.050138    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0" containerName="revoke-keys"
Nov 02 02:30:14.050269 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.050254    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0" containerName="revoke-keys"
Nov 02 02:30:14.053606 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.053588    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.111150 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.111111    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-18df32-openshift-default-dockercfg-mxxt4\""
Nov 02 02:30:14.140785 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.140751    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"]
Nov 02 02:30:14.225370 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.225326    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-workload-socket\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.225689 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.225668    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f495413-6902-46f8-99b8-866cda05a5d5-istio-token\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.225839 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.225824    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f495413-6902-46f8-99b8-866cda05a5d5-istio-podinfo\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.226029 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.226014    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-istio-envoy\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.226367 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.226332    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f495413-6902-46f8-99b8-866cda05a5d5-istiod-ca-cert\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.226531 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.226381    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6gg8\" (UniqueName: \"kubernetes.io/projected/7f495413-6902-46f8-99b8-866cda05a5d5-kube-api-access-m6gg8\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.226531 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.226417    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-credential-socket\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.226531 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.226445    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-istio-data\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.226531 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.226489    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-workload-certs\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.328421 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.328340    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f495413-6902-46f8-99b8-866cda05a5d5-istiod-ca-cert\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.328421 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.328380    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6gg8\" (UniqueName: \"kubernetes.io/projected/7f495413-6902-46f8-99b8-866cda05a5d5-kube-api-access-m6gg8\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.328421 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.328402    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-credential-socket\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.328421 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.328421    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-istio-data\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.328792 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.328455    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-workload-certs\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.328792 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.328492    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-workload-socket\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.328792 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.328538    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f495413-6902-46f8-99b8-866cda05a5d5-istio-token\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.328792 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.328574    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f495413-6902-46f8-99b8-866cda05a5d5-istio-podinfo\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.328792 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.328637    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-istio-envoy\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.329085 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.328871    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-credential-socket\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.329085 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.328944    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-workload-certs\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.329085 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.329028    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-istio-data\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.329248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.329106    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f495413-6902-46f8-99b8-866cda05a5d5-istiod-ca-cert\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.329248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.329199    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-workload-socket\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.331087 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.331062    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-istio-envoy\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.331522 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.331503    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f495413-6902-46f8-99b8-866cda05a5d5-istio-podinfo\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.389944 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.389900    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f495413-6902-46f8-99b8-866cda05a5d5-istio-token\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.390094 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.389922    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6gg8\" (UniqueName: \"kubernetes.io/projected/7f495413-6902-46f8-99b8-866cda05a5d5-kube-api-access-m6gg8\") pod \"e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") " pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:14.673433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:14.673342    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:15.104858 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:15.104820    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"]
Nov 02 02:30:15.106863 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:30:15.106832    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f495413_6902_46f8_99b8_866cda05a5d5.slice/crio-16119e1aeef9b9c7c1c0f2b07ae96d494e0e4f62adaf3703a03f845f34c111ee WatchSource:0}: Error finding container 16119e1aeef9b9c7c1c0f2b07ae96d494e0e4f62adaf3703a03f845f34c111ee: Status 404 returned error can't find the container with id 16119e1aeef9b9c7c1c0f2b07ae96d494e0e4f62adaf3703a03f845f34c111ee
Nov 02 02:30:15.109162 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:15.109126    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:30:15.109308 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:15.109209    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:30:15.109308 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:15.109272    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:30:15.759983 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:15.759946    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m" event={"ID":"7f495413-6902-46f8-99b8-866cda05a5d5","Type":"ContainerStarted","Data":"b9eaca34c406193bfd3ce42d087333924b2053244a8dde96c965372d33f423eb"}
Nov 02 02:30:15.760182 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:15.759997    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m" event={"ID":"7f495413-6902-46f8-99b8-866cda05a5d5","Type":"ContainerStarted","Data":"16119e1aeef9b9c7c1c0f2b07ae96d494e0e4f62adaf3703a03f845f34c111ee"}
Nov 02 02:30:15.931157 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:15.931104    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m" podStartSLOduration=2.9310896140000002 podStartE2EDuration="2.931089614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:30:15.930106986 +0000 UTC m=+3463.371143483" watchObservedRunningTime="2026-11-02 02:30:15.931089614 +0000 UTC m=+3463.372126110"
Nov 02 02:30:16.673662 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:16.673636    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:16.678876 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:16.678845    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:16.765476 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:16.765438    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:16.766568 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:16.766544    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:31.407962 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:31.407925    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:30:41.325820 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:30:41.325777    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:30:41.901917 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:41.901884    5875 generic.go:356] "Generic (PLEG): container finished" podID="7f495413-6902-46f8-99b8-866cda05a5d5" containerID="b9eaca34c406193bfd3ce42d087333924b2053244a8dde96c965372d33f423eb" exitCode=0
Nov 02 02:30:41.902108 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:41.901962    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m" event={"ID":"7f495413-6902-46f8-99b8-866cda05a5d5","Type":"ContainerDied","Data":"b9eaca34c406193bfd3ce42d087333924b2053244a8dde96c965372d33f423eb"}
Nov 02 02:30:41.902437 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:41.902421    5875 scope.go:122] "RemoveContainer" containerID="b9eaca34c406193bfd3ce42d087333924b2053244a8dde96c965372d33f423eb"
Nov 02 02:30:41.903768 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:41.903739    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:30:41.903832 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:41.903809    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:30:41.903868 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:41.903840    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:30:42.927959 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:42.927927    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m" event={"ID":"7f495413-6902-46f8-99b8-866cda05a5d5","Type":"ContainerStarted","Data":"b975327e811397babe081bef663beb5941a3b27aed9552ddff417f1acb983921"}
Nov 02 02:30:43.674573 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:43.674535    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:43.675741 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:43.675710    5875 patch_prober.go:29] interesting pod/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body=
Nov 02 02:30:43.675869 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:43.675761    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m" podUID="7f495413-6902-46f8-99b8-866cda05a5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused"
Nov 02 02:30:44.674140 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:44.674101    5875 patch_prober.go:29] interesting pod/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body=
Nov 02 02:30:44.674662 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:44.674171    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m" podUID="7f495413-6902-46f8-99b8-866cda05a5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused"
Nov 02 02:30:45.674702 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:45.674666    5875 patch_prober.go:29] interesting pod/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body=
Nov 02 02:30:45.675131 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:45.674731    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m" podUID="7f495413-6902-46f8-99b8-866cda05a5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused"
Nov 02 02:30:46.675676 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:46.673769    5875 patch_prober.go:29] interesting pod/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body=
Nov 02 02:30:46.675676 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:46.673842    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m" podUID="7f495413-6902-46f8-99b8-866cda05a5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused"
Nov 02 02:30:46.766313 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:46.766282    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:47.674604 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:47.674568    5875 patch_prober.go:29] interesting pod/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body=
Nov 02 02:30:47.674791 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:47.674620    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m" podUID="7f495413-6902-46f8-99b8-866cda05a5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused"
Nov 02 02:30:48.678117 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:48.678089    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:48.679095 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:48.679075    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:30:51.601990 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:30:51.601957    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:31:11.317961 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:31:11.317873    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:31:17.662742 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:17.662681    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:31:32.649890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:32.649859    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28798c31-qgz27"]
Nov 02 02:31:32.653625 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:32.653607    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28798c31-qgz27"
Nov 02 02:31:32.705659 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:32.705636    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:31:32.705785 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:32.705701    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:31:32.756103 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:32.756075    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:31:32.756302 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:32.756144    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:31:32.757428 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:32.757401    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql5ll\" (UniqueName: \"kubernetes.io/projected/c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3-kube-api-access-ql5ll\") pod \"maas-api-revoke-sub-28798c31-qgz27\" (UID: \"c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28798c31-qgz27"
Nov 02 02:31:32.757541 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:32.757453    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28798c31-qgz27\" (UID: \"c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28798c31-qgz27"
Nov 02 02:31:32.757615 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:32.757596    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28798c31-qgz27\" (UID: \"c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28798c31-qgz27"
Nov 02 02:31:32.757957 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:32.757938    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28798c31-qgz27"]
Nov 02 02:31:32.859210 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:32.859173    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28798c31-qgz27\" (UID: \"c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28798c31-qgz27"
Nov 02 02:31:32.859432 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:32.859363    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql5ll\" (UniqueName: \"kubernetes.io/projected/c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3-kube-api-access-ql5ll\") pod \"maas-api-revoke-sub-28798c31-qgz27\" (UID: \"c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28798c31-qgz27"
Nov 02 02:31:32.859432 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:32.859406    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28798c31-qgz27\" (UID: \"c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28798c31-qgz27"
Nov 02 02:31:32.859773 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:32.859749    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28798c31-qgz27\" (UID: \"c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28798c31-qgz27"
Nov 02 02:31:32.861762 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:32.861742    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28798c31-qgz27\" (UID: \"c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28798c31-qgz27"
Nov 02 02:31:32.920395 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:32.920339    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql5ll\" (UniqueName: \"kubernetes.io/projected/c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3-kube-api-access-ql5ll\") pod \"maas-api-revoke-sub-28798c31-qgz27\" (UID: \"c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28798c31-qgz27"
Nov 02 02:31:32.966730 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:32.966708    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28798c31-qgz27"
Nov 02 02:31:33.337124 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:33.337100    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28798c31-qgz27"]
Nov 02 02:31:33.339341 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:31:33.339308    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e835c0_7fb1_43ff_863a_ceef2e3ca8e3.slice/crio-4bf94975314955847101b82e9d34596bb0c03d4529b14805f6c35a191bc07df1 WatchSource:0}: Error finding container 4bf94975314955847101b82e9d34596bb0c03d4529b14805f6c35a191bc07df1: Status 404 returned error can't find the container with id 4bf94975314955847101b82e9d34596bb0c03d4529b14805f6c35a191bc07df1
Nov 02 02:31:34.190540 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:34.190507    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28798c31-qgz27" event={"ID":"c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3","Type":"ContainerStarted","Data":"cac45eff0393c0500e3fcd9d846a3005ae71f9c752762af9f3f8fbacbe5bb8d9"}
Nov 02 02:31:34.190967 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:34.190557    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28798c31-qgz27" event={"ID":"c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3","Type":"ContainerStarted","Data":"4bf94975314955847101b82e9d34596bb0c03d4529b14805f6c35a191bc07df1"}
Nov 02 02:31:34.305059 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:34.305004    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28798c31-qgz27" podStartSLOduration=2.30499073 podStartE2EDuration="2.30499073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:31:34.304099207 +0000 UTC m=+3541.745135718" watchObservedRunningTime="2026-11-02 02:31:34.30499073 +0000 UTC m=+3541.746027227"
Nov 02 02:31:35.196844 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:35.196808    5875 generic.go:356] "Generic (PLEG): container finished" podID="c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3" containerID="cac45eff0393c0500e3fcd9d846a3005ae71f9c752762af9f3f8fbacbe5bb8d9" exitCode=0
Nov 02 02:31:35.197289 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:35.196863    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28798c31-qgz27" event={"ID":"c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3","Type":"ContainerDied","Data":"cac45eff0393c0500e3fcd9d846a3005ae71f9c752762af9f3f8fbacbe5bb8d9"}
Nov 02 02:31:36.348385 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:36.348361    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28798c31-qgz27"
Nov 02 02:31:36.495385 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:36.495356    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3-kube-api-access-ql5ll\" (UniqueName: \"kubernetes.io/projected/c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3-kube-api-access-ql5ll\") pod \"c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3\" (UID: \"c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3\") "
Nov 02 02:31:36.495597 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:36.495391    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3-maas-api-openshift-ca\") pod \"c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3\" (UID: \"c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3\") "
Nov 02 02:31:36.495597 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:36.495440    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3-maas-api-xks-ca\") pod \"c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3\" (UID: \"c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3\") "
Nov 02 02:31:36.495807 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:36.495785    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3-maas-api-openshift-ca" pod "c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3" (UID: "c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:31:36.496498 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:36.496468    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:31:36.499391 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:36.499360    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3-kube-api-access-ql5ll" pod "c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3" (UID: "c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3"). InnerVolumeSpecName "kube-api-access-ql5ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:31:36.506326 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:36.506302    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3-maas-api-xks-ca" pod "c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3" (UID: "c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:31:36.597202 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:36.597172    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql5ll\" (UniqueName: \"kubernetes.io/projected/c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3-kube-api-access-ql5ll\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:31:36.597202 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:36.597198    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:31:37.207863 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:37.207833    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28798c31-qgz27"
Nov 02 02:31:37.208046 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:37.207866    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28798c31-qgz27" event={"ID":"c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3","Type":"ContainerDied","Data":"4bf94975314955847101b82e9d34596bb0c03d4529b14805f6c35a191bc07df1"}
Nov 02 02:31:37.208046 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:37.207918    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bf94975314955847101b82e9d34596bb0c03d4529b14805f6c35a191bc07df1"
Nov 02 02:31:41.317425 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:31:41.317388    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:31:44.292697 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:44.292654    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9"]
Nov 02 02:31:44.293194 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:44.293140    5875 cpu_manager.go:414] "containerMap: removing container" podUID="c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3" containerName="revoke-keys"
Nov 02 02:31:44.293194 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:44.293155    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3" containerName="revoke-keys"
Nov 02 02:31:44.293363 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:44.293284    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3" containerName="revoke-keys"
Nov 02 02:31:44.296359 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:44.296335    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9"
Nov 02 02:31:44.352114 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:44.352075    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9"]
Nov 02 02:31:44.398647 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:44.398603    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 02:31:44.398647 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:44.398618    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:31:44.398647 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:44.398648    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:31:44.398917 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:44.398701    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnwgr\""
Nov 02 02:31:44.473874 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:44.473837    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e67f99a-d4b5-4bbc-a131-641187367a36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9\" (UID: \"8e67f99a-d4b5-4bbc-a131-641187367a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9"
Nov 02 02:31:44.474066 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:44.473892    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmn8c\" (UniqueName: \"kubernetes.io/projected/8e67f99a-d4b5-4bbc-a131-641187367a36-kube-api-access-rmn8c\") pod \"maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9\" (UID: \"8e67f99a-d4b5-4bbc-a131-641187367a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9"
Nov 02 02:31:44.474066 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:44.473916    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e67f99a-d4b5-4bbc-a131-641187367a36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9\" (UID: \"8e67f99a-d4b5-4bbc-a131-641187367a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9"
Nov 02 02:31:44.575150 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:44.575061    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmn8c\" (UniqueName: \"kubernetes.io/projected/8e67f99a-d4b5-4bbc-a131-641187367a36-kube-api-access-rmn8c\") pod \"maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9\" (UID: \"8e67f99a-d4b5-4bbc-a131-641187367a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9"
Nov 02 02:31:44.575150 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:44.575103    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e67f99a-d4b5-4bbc-a131-641187367a36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9\" (UID: \"8e67f99a-d4b5-4bbc-a131-641187367a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9"
Nov 02 02:31:44.575459 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:44.575251    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e67f99a-d4b5-4bbc-a131-641187367a36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9\" (UID: \"8e67f99a-d4b5-4bbc-a131-641187367a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9"
Nov 02 02:31:44.575901 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:44.575881    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e67f99a-d4b5-4bbc-a131-641187367a36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9\" (UID: \"8e67f99a-d4b5-4bbc-a131-641187367a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9"
Nov 02 02:31:44.577691 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:44.577673    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e67f99a-d4b5-4bbc-a131-641187367a36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9\" (UID: \"8e67f99a-d4b5-4bbc-a131-641187367a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9"
Nov 02 02:31:44.635990 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:44.635965    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmn8c\" (UniqueName: \"kubernetes.io/projected/8e67f99a-d4b5-4bbc-a131-641187367a36-kube-api-access-rmn8c\") pod \"maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9\" (UID: \"8e67f99a-d4b5-4bbc-a131-641187367a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9"
Nov 02 02:31:44.910773 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:44.910700    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9"
Nov 02 02:31:45.283197 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:45.283158    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9"]
Nov 02 02:31:45.286276 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:31:45.286215    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e67f99a_d4b5_4bbc_a131_641187367a36.slice/crio-7f92e4973ad6fac8026cbe1b4b8a3cb3d860fa664d775de73573891503b10522 WatchSource:0}: Error finding container 7f92e4973ad6fac8026cbe1b4b8a3cb3d860fa664d775de73573891503b10522: Status 404 returned error can't find the container with id 7f92e4973ad6fac8026cbe1b4b8a3cb3d860fa664d775de73573891503b10522
Nov 02 02:31:45.801547 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:45.801516    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:31:46.256756 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:46.256723    5875 generic.go:356] "Generic (PLEG): container finished" podID="8e67f99a-d4b5-4bbc-a131-641187367a36" containerID="06cf501488ab21bab39307a543f0b4318202e250576ee642571c2688f7d37e1a" exitCode=0
Nov 02 02:31:46.256951 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:46.256789    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9" event={"ID":"8e67f99a-d4b5-4bbc-a131-641187367a36","Type":"ContainerDied","Data":"06cf501488ab21bab39307a543f0b4318202e250576ee642571c2688f7d37e1a"}
Nov 02 02:31:46.256951 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:46.256827    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9" event={"ID":"8e67f99a-d4b5-4bbc-a131-641187367a36","Type":"ContainerStarted","Data":"7f92e4973ad6fac8026cbe1b4b8a3cb3d860fa664d775de73573891503b10522"}
Nov 02 02:31:47.400584 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:47.400558    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9"
Nov 02 02:31:47.503406 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:47.503373    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e67f99a-d4b5-4bbc-a131-641187367a36-kube-api-access-rmn8c\" (UniqueName: \"kubernetes.io/projected/8e67f99a-d4b5-4bbc-a131-641187367a36-kube-api-access-rmn8c\") pod \"8e67f99a-d4b5-4bbc-a131-641187367a36\" (UID: \"8e67f99a-d4b5-4bbc-a131-641187367a36\") "
Nov 02 02:31:47.503619 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:47.503463    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e67f99a-d4b5-4bbc-a131-641187367a36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e67f99a-d4b5-4bbc-a131-641187367a36-maas-api-xks-ca\") pod \"8e67f99a-d4b5-4bbc-a131-641187367a36\" (UID: \"8e67f99a-d4b5-4bbc-a131-641187367a36\") "
Nov 02 02:31:47.503619 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:47.503570    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e67f99a-d4b5-4bbc-a131-641187367a36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e67f99a-d4b5-4bbc-a131-641187367a36-maas-api-openshift-ca\") pod \"8e67f99a-d4b5-4bbc-a131-641187367a36\" (UID: \"8e67f99a-d4b5-4bbc-a131-641187367a36\") "
Nov 02 02:31:47.504034 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:47.503997    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e67f99a-d4b5-4bbc-a131-641187367a36-maas-api-openshift-ca" pod "8e67f99a-d4b5-4bbc-a131-641187367a36" (UID: "8e67f99a-d4b5-4bbc-a131-641187367a36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:31:47.506154 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:47.506129    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e67f99a-d4b5-4bbc-a131-641187367a36-maas-api-xks-ca" pod "8e67f99a-d4b5-4bbc-a131-641187367a36" (UID: "8e67f99a-d4b5-4bbc-a131-641187367a36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:31:47.506393 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:47.506363    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e67f99a-d4b5-4bbc-a131-641187367a36-kube-api-access-rmn8c" pod "8e67f99a-d4b5-4bbc-a131-641187367a36" (UID: "8e67f99a-d4b5-4bbc-a131-641187367a36"). InnerVolumeSpecName "kube-api-access-rmn8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:31:47.605563 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:47.605522    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e67f99a-d4b5-4bbc-a131-641187367a36-maas-api-xks-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:31:47.605563 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:47.605555    5875 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e67f99a-d4b5-4bbc-a131-641187367a36-maas-api-openshift-ca\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:31:47.605563 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:47.605566    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmn8c\" (UniqueName: \"kubernetes.io/projected/8e67f99a-d4b5-4bbc-a131-641187367a36-kube-api-access-rmn8c\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:31:48.267747 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:48.267715    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9"
Nov 02 02:31:48.267921 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:48.267749    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9" event={"ID":"8e67f99a-d4b5-4bbc-a131-641187367a36","Type":"ContainerDied","Data":"7f92e4973ad6fac8026cbe1b4b8a3cb3d860fa664d775de73573891503b10522"}
Nov 02 02:31:48.267921 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:31:48.267799    5875 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f92e4973ad6fac8026cbe1b4b8a3cb3d860fa664d775de73573891503b10522"
Nov 02 02:32:02.046390 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:02.046355    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3b32f86-h6swk"]
Nov 02 02:32:02.079281 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:02.079254    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3b32f86-h6swk"]
Nov 02 02:32:02.693514 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:02.693481    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569" path="/var/lib/kubelet/pods/22b39ba1-7a7d-4fb1-bb18-b0bdb36b8569/volumes"
Nov 02 02:32:04.489731 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:04.489692    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9"]
Nov 02 02:32:04.539324 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:04.539292    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-18df32-kvlv9"]
Nov 02 02:32:04.692855 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:04.692823    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e67f99a-d4b5-4bbc-a131-641187367a36" path="/var/lib/kubelet/pods/8e67f99a-d4b5-4bbc-a131-641187367a36/volumes"
Nov 02 02:32:06.991672 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:06.991625    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:32:11.321306 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:32:11.321269    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:32:11.745937 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:11.745904    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"]
Nov 02 02:32:11.746116 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:11.746098    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m" podUID="7f495413-6902-46f8-99b8-866cda05a5d5" containerName="istio-proxy" containerID="cri-o://b975327e811397babe081bef663beb5941a3b27aed9552ddff417f1acb983921" gracePeriod=30
Nov 02 02:32:17.006670 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.006640    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:32:17.137361 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.137281    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-credential-socket\") pod \"7f495413-6902-46f8-99b8-866cda05a5d5\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") "
Nov 02 02:32:17.137361 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.137329    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-istio-data\") pod \"7f495413-6902-46f8-99b8-866cda05a5d5\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") "
Nov 02 02:32:17.137361 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.137348    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-workload-socket\") pod \"7f495413-6902-46f8-99b8-866cda05a5d5\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") "
Nov 02 02:32:17.137668 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.137404    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f495413-6902-46f8-99b8-866cda05a5d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f495413-6902-46f8-99b8-866cda05a5d5-istiod-ca-cert\") pod \"7f495413-6902-46f8-99b8-866cda05a5d5\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") "
Nov 02 02:32:17.137668 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.137422    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-istio-envoy\") pod \"7f495413-6902-46f8-99b8-866cda05a5d5\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") "
Nov 02 02:32:17.137668 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.137469    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f495413-6902-46f8-99b8-866cda05a5d5-kube-api-access-m6gg8\" (UniqueName: \"kubernetes.io/projected/7f495413-6902-46f8-99b8-866cda05a5d5-kube-api-access-m6gg8\") pod \"7f495413-6902-46f8-99b8-866cda05a5d5\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") "
Nov 02 02:32:17.137668 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.137550    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f495413-6902-46f8-99b8-866cda05a5d5-istio-token\" (UniqueName: \"kubernetes.io/projected/7f495413-6902-46f8-99b8-866cda05a5d5-istio-token\") pod \"7f495413-6902-46f8-99b8-866cda05a5d5\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") "
Nov 02 02:32:17.137668 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.137617    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f495413-6902-46f8-99b8-866cda05a5d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f495413-6902-46f8-99b8-866cda05a5d5-istio-podinfo\") pod \"7f495413-6902-46f8-99b8-866cda05a5d5\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") "
Nov 02 02:32:17.137668 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.137657    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-credential-socket" pod "7f495413-6902-46f8-99b8-866cda05a5d5" (UID: "7f495413-6902-46f8-99b8-866cda05a5d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:32:17.137976 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.137665    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-workload-certs\") pod \"7f495413-6902-46f8-99b8-866cda05a5d5\" (UID: \"7f495413-6902-46f8-99b8-866cda05a5d5\") "
Nov 02 02:32:17.137976 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.137740    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-workload-socket" pod "7f495413-6902-46f8-99b8-866cda05a5d5" (UID: "7f495413-6902-46f8-99b8-866cda05a5d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:32:17.137976 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.137872    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f495413-6902-46f8-99b8-866cda05a5d5-istiod-ca-cert" pod "7f495413-6902-46f8-99b8-866cda05a5d5" (UID: "7f495413-6902-46f8-99b8-866cda05a5d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:32:17.137976 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.137882    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-workload-certs" pod "7f495413-6902-46f8-99b8-866cda05a5d5" (UID: "7f495413-6902-46f8-99b8-866cda05a5d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:32:17.137976 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.137602    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-istio-data" pod "7f495413-6902-46f8-99b8-866cda05a5d5" (UID: "7f495413-6902-46f8-99b8-866cda05a5d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:32:17.138339 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.138245    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:17.138339 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.138273    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:17.138339 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.138286    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:17.138339 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.138300    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f495413-6902-46f8-99b8-866cda05a5d5-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:17.138339 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.138315    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:17.140200 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.140108    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f495413-6902-46f8-99b8-866cda05a5d5-kube-api-access-m6gg8" pod "7f495413-6902-46f8-99b8-866cda05a5d5" (UID: "7f495413-6902-46f8-99b8-866cda05a5d5"). InnerVolumeSpecName "kube-api-access-m6gg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:32:17.140348 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.140257    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f495413-6902-46f8-99b8-866cda05a5d5-istio-token" pod "7f495413-6902-46f8-99b8-866cda05a5d5" (UID: "7f495413-6902-46f8-99b8-866cda05a5d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:32:17.140555 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.140533    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f495413-6902-46f8-99b8-866cda05a5d5-istio-podinfo" pod "7f495413-6902-46f8-99b8-866cda05a5d5" (UID: "7f495413-6902-46f8-99b8-866cda05a5d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:32:17.140602 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.140553    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-istio-envoy" pod "7f495413-6902-46f8-99b8-866cda05a5d5" (UID: "7f495413-6902-46f8-99b8-866cda05a5d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:32:17.240328 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.239618    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f495413-6902-46f8-99b8-866cda05a5d5-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:17.240328 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.240155    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6gg8\" (UniqueName: \"kubernetes.io/projected/7f495413-6902-46f8-99b8-866cda05a5d5-kube-api-access-m6gg8\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:17.240328 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.240176    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f495413-6902-46f8-99b8-866cda05a5d5-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:17.240328 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.240192    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f495413-6902-46f8-99b8-866cda05a5d5-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:32:17.443298 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.443186    5875 generic.go:356] "Generic (PLEG): container finished" podID="7f495413-6902-46f8-99b8-866cda05a5d5" containerID="b975327e811397babe081bef663beb5941a3b27aed9552ddff417f1acb983921" exitCode=0
Nov 02 02:32:17.443298 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.443268    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m" event={"ID":"7f495413-6902-46f8-99b8-866cda05a5d5","Type":"ContainerDied","Data":"b975327e811397babe081bef663beb5941a3b27aed9552ddff417f1acb983921"}
Nov 02 02:32:17.443298 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.443293    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"
Nov 02 02:32:17.443629 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.443362    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m" event={"ID":"7f495413-6902-46f8-99b8-866cda05a5d5","Type":"ContainerDied","Data":"16119e1aeef9b9c7c1c0f2b07ae96d494e0e4f62adaf3703a03f845f34c111ee"}
Nov 02 02:32:17.443629 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.443388    5875 scope.go:122] "RemoveContainer" containerID="b975327e811397babe081bef663beb5941a3b27aed9552ddff417f1acb983921"
Nov 02 02:32:17.454007 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.453977    5875 scope.go:122] "RemoveContainer" containerID="b9eaca34c406193bfd3ce42d087333924b2053244a8dde96c965372d33f423eb"
Nov 02 02:32:17.462990 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.462966    5875 scope.go:122] "RemoveContainer" containerID="b975327e811397babe081bef663beb5941a3b27aed9552ddff417f1acb983921"
Nov 02 02:32:17.463361 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:32:17.463309    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b975327e811397babe081bef663beb5941a3b27aed9552ddff417f1acb983921\": container with ID starting with b975327e811397babe081bef663beb5941a3b27aed9552ddff417f1acb983921 not found: ID does not exist" containerID="b975327e811397babe081bef663beb5941a3b27aed9552ddff417f1acb983921"
Nov 02 02:32:17.463454 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.463370    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b975327e811397babe081bef663beb5941a3b27aed9552ddff417f1acb983921"} err="failed to get container status \"b975327e811397babe081bef663beb5941a3b27aed9552ddff417f1acb983921\": rpc error: code = NotFound desc = could not find container \"b975327e811397babe081bef663beb5941a3b27aed9552ddff417f1acb983921\": container with ID starting with b975327e811397babe081bef663beb5941a3b27aed9552ddff417f1acb983921 not found: ID does not exist"
Nov 02 02:32:17.463454 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.463390    5875 scope.go:122] "RemoveContainer" containerID="b9eaca34c406193bfd3ce42d087333924b2053244a8dde96c965372d33f423eb"
Nov 02 02:32:17.463675 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:32:17.463653    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9eaca34c406193bfd3ce42d087333924b2053244a8dde96c965372d33f423eb\": container with ID starting with b9eaca34c406193bfd3ce42d087333924b2053244a8dde96c965372d33f423eb not found: ID does not exist" containerID="b9eaca34c406193bfd3ce42d087333924b2053244a8dde96c965372d33f423eb"
Nov 02 02:32:17.463723 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.463682    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9eaca34c406193bfd3ce42d087333924b2053244a8dde96c965372d33f423eb"} err="failed to get container status \"b9eaca34c406193bfd3ce42d087333924b2053244a8dde96c965372d33f423eb\": rpc error: code = NotFound desc = could not find container \"b9eaca34c406193bfd3ce42d087333924b2053244a8dde96c965372d33f423eb\": container with ID starting with b9eaca34c406193bfd3ce42d087333924b2053244a8dde96c965372d33f423eb not found: ID does not exist"
Nov 02 02:32:17.606750 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.606718    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"]
Nov 02 02:32:17.656359 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:17.656328    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-18df32-openshift-default-6fff9f6c576pg9m"]
Nov 02 02:32:18.694682 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:18.694648    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f495413-6902-46f8-99b8-866cda05a5d5" path="/var/lib/kubelet/pods/7f495413-6902-46f8-99b8-866cda05a5d5/volumes"
Nov 02 02:32:34.972532 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:32:34.972491    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:32:41.317675 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:32:41.317624    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:33:01.417234 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:33:01.417194    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:33:11.318079 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:33:11.318041    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:33:27.080827 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:33:27.080791    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:33:29.620247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:33:29.619378    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-mfmmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:33:29.620247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:33:29.619466    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-mfmmj" podUID="004e401a-4d50-46ec-b1bd-d9896f9b20d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:33:39.789149 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:33:39.789112    5875 scope.go:122] "RemoveContainer" containerID="55a9529d636fc271a871732aed393fadc2227eb2899c60bb3f8a7e8f527450c7"
Nov 02 02:33:39.797764 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:33:39.797740    5875 scope.go:122] "RemoveContainer" containerID="f16db435dd06551afd9afd70dc9d43d8bd57bfd4e23fbc6ce5ee90f3b196bf0d"
Nov 02 02:33:41.318275 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:33:41.318232    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:33:42.610907 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:33:42.610862    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:33:56.110961 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:33:56.110920    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:34:05.181451 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:05.181401    5875 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ghslr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:34:05.182253 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:05.182201    5875 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ghslr" podUID="0c2d802d-8fbd-430d-ae72-29b0e217714d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:34:07.762696 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:07.762664    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:34:11.316951 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:34:11.316855    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:34:28.600618 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:28.600575    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:34:29.033325 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.033287    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d4c89cd86-fcs9s"]
Nov 02 02:34:29.033982 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.033965    5875 cpu_manager.go:414] "containerMap: removing container" podUID="8e67f99a-d4b5-4bbc-a131-641187367a36" containerName="revoke-keys"
Nov 02 02:34:29.034098 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.033984    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e67f99a-d4b5-4bbc-a131-641187367a36" containerName="revoke-keys"
Nov 02 02:34:29.034098 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.034010    5875 cpu_manager.go:414] "containerMap: removing container" podUID="7f495413-6902-46f8-99b8-866cda05a5d5" containerName="istio-proxy"
Nov 02 02:34:29.034098 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.034018    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f495413-6902-46f8-99b8-866cda05a5d5" containerName="istio-proxy"
Nov 02 02:34:29.034098 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.034052    5875 cpu_manager.go:414] "containerMap: removing container" podUID="7f495413-6902-46f8-99b8-866cda05a5d5" containerName="istio-proxy"
Nov 02 02:34:29.034098 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.034059    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f495413-6902-46f8-99b8-866cda05a5d5" containerName="istio-proxy"
Nov 02 02:34:29.034401 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.034176    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e67f99a-d4b5-4bbc-a131-641187367a36" containerName="revoke-keys"
Nov 02 02:34:29.034401 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.034193    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f495413-6902-46f8-99b8-866cda05a5d5" containerName="istio-proxy"
Nov 02 02:34:29.034401 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.034207    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f495413-6902-46f8-99b8-866cda05a5d5" containerName="istio-proxy"
Nov 02 02:34:29.039211 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.039181    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-fcs9s"
Nov 02 02:34:29.134353 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.134187    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7381a4a-d748-49f1-993c-cb34b0254e49-metrics-certs\") pod \"maas-api-6d4c89cd86-fcs9s\" (UID: \"a7381a4a-d748-49f1-993c-cb34b0254e49\") " pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-fcs9s"
Nov 02 02:34:29.134525 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.134359    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlzrr\" (UniqueName: \"kubernetes.io/projected/a7381a4a-d748-49f1-993c-cb34b0254e49-kube-api-access-mlzrr\") pod \"maas-api-6d4c89cd86-fcs9s\" (UID: \"a7381a4a-d748-49f1-993c-cb34b0254e49\") " pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-fcs9s"
Nov 02 02:34:29.134525 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.134396    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7381a4a-d748-49f1-993c-cb34b0254e49-maas-api-tls\") pod \"maas-api-6d4c89cd86-fcs9s\" (UID: \"a7381a4a-d748-49f1-993c-cb34b0254e49\") " pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-fcs9s"
Nov 02 02:34:29.134627 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.134519    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7381a4a-d748-49f1-993c-cb34b0254e49-openshift-service-ca\") pod \"maas-api-6d4c89cd86-fcs9s\" (UID: \"a7381a4a-d748-49f1-993c-cb34b0254e49\") " pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-fcs9s"
Nov 02 02:34:29.141088 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.141062    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 02:34:29.141259 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.141089    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 02:34:29.141259 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.141176    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 02 02:34:29.141259 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.141192    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 02:34:29.141652 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.141296    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2n4c5\""
Nov 02 02:34:29.143062 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.143043    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d4c89cd86-fcs9s"]
Nov 02 02:34:29.235720 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.235673    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlzrr\" (UniqueName: \"kubernetes.io/projected/a7381a4a-d748-49f1-993c-cb34b0254e49-kube-api-access-mlzrr\") pod \"maas-api-6d4c89cd86-fcs9s\" (UID: \"a7381a4a-d748-49f1-993c-cb34b0254e49\") " pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-fcs9s"
Nov 02 02:34:29.235923 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.235732    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7381a4a-d748-49f1-993c-cb34b0254e49-maas-api-tls\") pod \"maas-api-6d4c89cd86-fcs9s\" (UID: \"a7381a4a-d748-49f1-993c-cb34b0254e49\") " pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-fcs9s"
Nov 02 02:34:29.235923 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.235794    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7381a4a-d748-49f1-993c-cb34b0254e49-openshift-service-ca\") pod \"maas-api-6d4c89cd86-fcs9s\" (UID: \"a7381a4a-d748-49f1-993c-cb34b0254e49\") " pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-fcs9s"
Nov 02 02:34:29.235923 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.235867    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7381a4a-d748-49f1-993c-cb34b0254e49-metrics-certs\") pod \"maas-api-6d4c89cd86-fcs9s\" (UID: \"a7381a4a-d748-49f1-993c-cb34b0254e49\") " pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-fcs9s"
Nov 02 02:34:29.236732 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.236644    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7381a4a-d748-49f1-993c-cb34b0254e49-openshift-service-ca\") pod \"maas-api-6d4c89cd86-fcs9s\" (UID: \"a7381a4a-d748-49f1-993c-cb34b0254e49\") " pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-fcs9s"
Nov 02 02:34:29.238426 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.238400    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7381a4a-d748-49f1-993c-cb34b0254e49-metrics-certs\") pod \"maas-api-6d4c89cd86-fcs9s\" (UID: \"a7381a4a-d748-49f1-993c-cb34b0254e49\") " pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-fcs9s"
Nov 02 02:34:29.238426 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.238416    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7381a4a-d748-49f1-993c-cb34b0254e49-maas-api-tls\") pod \"maas-api-6d4c89cd86-fcs9s\" (UID: \"a7381a4a-d748-49f1-993c-cb34b0254e49\") " pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-fcs9s"
Nov 02 02:34:29.297326 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.297257    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlzrr\" (UniqueName: \"kubernetes.io/projected/a7381a4a-d748-49f1-993c-cb34b0254e49-kube-api-access-mlzrr\") pod \"maas-api-6d4c89cd86-fcs9s\" (UID: \"a7381a4a-d748-49f1-993c-cb34b0254e49\") " pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-fcs9s"
Nov 02 02:34:29.378893 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.378849    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-fcs9s"
Nov 02 02:34:29.753595 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.753533    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d4c89cd86-fcs9s"]
Nov 02 02:34:29.756616 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:34:29.756578    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7381a4a_d748_49f1_993c_cb34b0254e49.slice/crio-a90e80796c80dd5cd07b70ab92ae28857daf61a8aa7c61746dd01466848e14b8 WatchSource:0}: Error finding container a90e80796c80dd5cd07b70ab92ae28857daf61a8aa7c61746dd01466848e14b8: Status 404 returned error can't find the container with id a90e80796c80dd5cd07b70ab92ae28857daf61a8aa7c61746dd01466848e14b8
Nov 02 02:34:29.757946 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:29.757927    5875 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 02:34:30.318061 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:30.318020    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-fcs9s" event={"ID":"a7381a4a-d748-49f1-993c-cb34b0254e49","Type":"ContainerStarted","Data":"a90e80796c80dd5cd07b70ab92ae28857daf61a8aa7c61746dd01466848e14b8"}
Nov 02 02:34:31.325353 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:31.325317    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-fcs9s" event={"ID":"a7381a4a-d748-49f1-993c-cb34b0254e49","Type":"ContainerStarted","Data":"3e5d88e0b23f925922494cb7dd867b73010e00fb0b6baa3dd87e16ed6ea4b0da"}
Nov 02 02:34:31.325839 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:31.325398    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-fcs9s"
Nov 02 02:34:31.443295 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:31.443239    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-fcs9s" podStartSLOduration=2.58714286 podStartE2EDuration="3.443212035s" totalImagesPullingTime="856.069175ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:34:31.441565292 +0000 UTC m=+3718.882601787" watchObservedRunningTime="2026-11-02 02:34:31.443212035 +0000 UTC m=+3718.884248544"
Nov 02 02:34:36.078965 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.078930    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"]
Nov 02 02:34:36.082680 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.082663    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.142251 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.137486    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-71e54d09-openshift-default-dockercfg-xxt8c\""
Nov 02 02:34:36.186139 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.186102    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"]
Nov 02 02:34:36.223242 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.223199    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-data\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.223437 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.223349    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcde29c5-e273-4dfa-9d3a-031331cb8482-istiod-ca-cert\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.223437 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.223379    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-workload-certs\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.223437 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.223396    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-envoy\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.223581 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.223522    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg4cm\" (UniqueName: \"kubernetes.io/projected/dcde29c5-e273-4dfa-9d3a-031331cb8482-kube-api-access-vg4cm\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.223621 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.223592    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-podinfo\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.223621 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.223612    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-token\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.223706 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.223630    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-workload-socket\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.223706 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.223652    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-credential-socket\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.324842 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.324808    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcde29c5-e273-4dfa-9d3a-031331cb8482-istiod-ca-cert\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.325054 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.324850    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-workload-certs\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.325054 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.324876    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-envoy\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.325054 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.324954    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg4cm\" (UniqueName: \"kubernetes.io/projected/dcde29c5-e273-4dfa-9d3a-031331cb8482-kube-api-access-vg4cm\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.325054 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.325011    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-podinfo\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.325054 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.325036    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-token\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.325395 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.325063    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-workload-socket\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.325395 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.325090    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-credential-socket\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.325395 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.325175    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-data\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.327286 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.326292    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcde29c5-e273-4dfa-9d3a-031331cb8482-istiod-ca-cert\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.327905 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.327854    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-workload-certs\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.328322 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.328273    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-credential-socket\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.328692 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.328658    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-workload-socket\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.329106 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.329050    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-data\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.332591 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.332565    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-podinfo\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.333772 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.333745    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-envoy\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.385894 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.385862    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-token\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.386076 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.386049    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg4cm\" (UniqueName: \"kubernetes.io/projected/dcde29c5-e273-4dfa-9d3a-031331cb8482-kube-api-access-vg4cm\") pod \"e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") " pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.395987 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.395962    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:36.821057 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.821035    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"]
Nov 02 02:34:36.823458 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:34:36.823431    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcde29c5_e273_4dfa_9d3a_031331cb8482.slice/crio-c88f99d8d9f47b686b0ce91e7f097d374c95e3d416f5669e971ffcf83ca5a7ec WatchSource:0}: Error finding container c88f99d8d9f47b686b0ce91e7f097d374c95e3d416f5669e971ffcf83ca5a7ec: Status 404 returned error can't find the container with id c88f99d8d9f47b686b0ce91e7f097d374c95e3d416f5669e971ffcf83ca5a7ec
Nov 02 02:34:36.825680 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.825650    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:34:36.825749 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.825711    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:34:36.825749 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:36.825738    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:34:37.337254 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:37.337191    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d4c89cd86-fcs9s"
Nov 02 02:34:37.362432 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:37.362398    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j" event={"ID":"dcde29c5-e273-4dfa-9d3a-031331cb8482","Type":"ContainerStarted","Data":"28f0fc622e1dcd72511016a97c7381512fe9df2ff096abd5f635c4f1293395a0"}
Nov 02 02:34:37.362590 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:37.362443    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j" event={"ID":"dcde29c5-e273-4dfa-9d3a-031331cb8482","Type":"ContainerStarted","Data":"c88f99d8d9f47b686b0ce91e7f097d374c95e3d416f5669e971ffcf83ca5a7ec"}
Nov 02 02:34:37.396799 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:37.396773    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:37.401559 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:37.401535    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:37.578027 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:37.577978    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j" podStartSLOduration=1.5779634470000001 podStartE2EDuration="1.577963447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:34:37.576030833 +0000 UTC m=+3725.017067328" watchObservedRunningTime="2026-11-02 02:34:37.577963447 +0000 UTC m=+3725.018999998"
Nov 02 02:34:38.368199 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:38.368169    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:38.369137 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:38.369114    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:34:41.319102 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:34:41.319062    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:34:52.767838 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:34:52.767796    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:35:02.503822 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:02.503791    5875 generic.go:356] "Generic (PLEG): container finished" podID="dcde29c5-e273-4dfa-9d3a-031331cb8482" containerID="28f0fc622e1dcd72511016a97c7381512fe9df2ff096abd5f635c4f1293395a0" exitCode=0
Nov 02 02:35:02.504534 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:02.503869    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j" event={"ID":"dcde29c5-e273-4dfa-9d3a-031331cb8482","Type":"ContainerDied","Data":"28f0fc622e1dcd72511016a97c7381512fe9df2ff096abd5f635c4f1293395a0"}
Nov 02 02:35:02.504534 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:02.504399    5875 scope.go:122] "RemoveContainer" containerID="28f0fc622e1dcd72511016a97c7381512fe9df2ff096abd5f635c4f1293395a0"
Nov 02 02:35:02.505747 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:02.505717    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:35:02.505822 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:02.505786    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:35:02.505822 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:02.505818    5875 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Nov 02 02:35:03.527083 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:03.527043    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j" event={"ID":"dcde29c5-e273-4dfa-9d3a-031331cb8482","Type":"ContainerStarted","Data":"875c1052440e0df06673f6675e51e809c76ffea08ead739e6738d1f7d20552f9"}
Nov 02 02:35:04.048611 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:04.048581    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-72fr5"]
Nov 02 02:35:04.051046 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:04.051014    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-72fr5"]
Nov 02 02:35:04.396671 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:04.396588    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:35:04.418247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:04.415365    5875 patch_prober.go:29] interesting pod/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 02 02:35:04.418247 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:04.415457    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j" podUID="dcde29c5-e273-4dfa-9d3a-031331cb8482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 02 02:35:04.693797 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:04.693708    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0" path="/var/lib/kubelet/pods/d1ebdd42-2ac4-4fc1-9934-445b4bfe12a0/volumes"
Nov 02 02:35:05.396975 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:05.396943    5875 patch_prober.go:29] interesting pod/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 02 02:35:05.397173 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:05.397003    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j" podUID="dcde29c5-e273-4dfa-9d3a-031331cb8482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 02 02:35:06.396507 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:06.396467    5875 patch_prober.go:29] interesting pod/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 02 02:35:06.396955 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:06.396536    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j" podUID="dcde29c5-e273-4dfa-9d3a-031331cb8482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 02 02:35:07.397195 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:07.397161    5875 patch_prober.go:29] interesting pod/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 02 02:35:07.397852 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:07.397262    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j" podUID="dcde29c5-e273-4dfa-9d3a-031331cb8482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 02 02:35:08.368927 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:08.368876    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:35:08.396775 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:08.396751    5875 patch_prober.go:29] interesting pod/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 02 02:35:08.396927 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:08.396794    5875 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j" podUID="dcde29c5-e273-4dfa-9d3a-031331cb8482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 02 02:35:09.400523 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:09.400493    5875 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:35:09.401538 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:09.401518    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:35:11.318594 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:35:11.318555    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:35:26.829136 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:26.829101    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"]
Nov 02 02:35:26.829655 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:26.829311    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j" podUID="dcde29c5-e273-4dfa-9d3a-031331cb8482" containerName="istio-proxy" containerID="cri-o://875c1052440e0df06673f6675e51e809c76ffea08ead739e6738d1f7d20552f9" gracePeriod=30
Nov 02 02:35:30.377214 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:30.377179    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:35:32.084991 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.084967    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:35:32.173373 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.173285    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-envoy\") pod \"dcde29c5-e273-4dfa-9d3a-031331cb8482\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") "
Nov 02 02:35:32.173373 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.173327    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-workload-certs\") pod \"dcde29c5-e273-4dfa-9d3a-031331cb8482\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") "
Nov 02 02:35:32.173622 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.173381    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-credential-socket\") pod \"dcde29c5-e273-4dfa-9d3a-031331cb8482\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") "
Nov 02 02:35:32.173622 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.173429    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-podinfo\") pod \"dcde29c5-e273-4dfa-9d3a-031331cb8482\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") "
Nov 02 02:35:32.173622 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.173463    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-workload-socket\") pod \"dcde29c5-e273-4dfa-9d3a-031331cb8482\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") "
Nov 02 02:35:32.173622 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.173497    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcde29c5-e273-4dfa-9d3a-031331cb8482-kube-api-access-vg4cm\" (UniqueName: \"kubernetes.io/projected/dcde29c5-e273-4dfa-9d3a-031331cb8482-kube-api-access-vg4cm\") pod \"dcde29c5-e273-4dfa-9d3a-031331cb8482\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") "
Nov 02 02:35:32.173622 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.173548    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-data\") pod \"dcde29c5-e273-4dfa-9d3a-031331cb8482\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") "
Nov 02 02:35:32.173890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.173625    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-token\" (UniqueName: \"kubernetes.io/projected/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-token\") pod \"dcde29c5-e273-4dfa-9d3a-031331cb8482\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") "
Nov 02 02:35:32.173890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.173654    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-workload-certs" pod "dcde29c5-e273-4dfa-9d3a-031331cb8482" (UID: "dcde29c5-e273-4dfa-9d3a-031331cb8482"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:35:32.173890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.173684    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcde29c5-e273-4dfa-9d3a-031331cb8482-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcde29c5-e273-4dfa-9d3a-031331cb8482-istiod-ca-cert\") pod \"dcde29c5-e273-4dfa-9d3a-031331cb8482\" (UID: \"dcde29c5-e273-4dfa-9d3a-031331cb8482\") "
Nov 02 02:35:32.173890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.173691    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-credential-socket" pod "dcde29c5-e273-4dfa-9d3a-031331cb8482" (UID: "dcde29c5-e273-4dfa-9d3a-031331cb8482"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:35:32.173890 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.173854    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-data" pod "dcde29c5-e273-4dfa-9d3a-031331cb8482" (UID: "dcde29c5-e273-4dfa-9d3a-031331cb8482"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:35:32.174170 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.173984    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-workload-socket" pod "dcde29c5-e273-4dfa-9d3a-031331cb8482" (UID: "dcde29c5-e273-4dfa-9d3a-031331cb8482"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:35:32.174170 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.174095    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcde29c5-e273-4dfa-9d3a-031331cb8482-istiod-ca-cert" pod "dcde29c5-e273-4dfa-9d3a-031331cb8482" (UID: "dcde29c5-e273-4dfa-9d3a-031331cb8482"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:35:32.174473 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.174343    5875 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-workload-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:32.174473 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.174366    5875 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-data\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:32.174473 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.174382    5875 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcde29c5-e273-4dfa-9d3a-031331cb8482-istiod-ca-cert\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:32.174473 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.174395    5875 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-workload-certs\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:32.174473 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.174412    5875 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-credential-socket\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:32.175842 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.175809    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-envoy" pod "dcde29c5-e273-4dfa-9d3a-031331cb8482" (UID: "dcde29c5-e273-4dfa-9d3a-031331cb8482"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:35:32.175965 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.175839    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcde29c5-e273-4dfa-9d3a-031331cb8482-kube-api-access-vg4cm" pod "dcde29c5-e273-4dfa-9d3a-031331cb8482" (UID: "dcde29c5-e273-4dfa-9d3a-031331cb8482"). InnerVolumeSpecName "kube-api-access-vg4cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:35:32.175965 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.175848    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-podinfo" pod "dcde29c5-e273-4dfa-9d3a-031331cb8482" (UID: "dcde29c5-e273-4dfa-9d3a-031331cb8482"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:35:32.175965 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.175884    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-token" pod "dcde29c5-e273-4dfa-9d3a-031331cb8482" (UID: "dcde29c5-e273-4dfa-9d3a-031331cb8482"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:35:32.275207 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.275172    5875 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-token\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:32.275207 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.275201    5875 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-envoy\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:32.275437 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.275251    5875 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcde29c5-e273-4dfa-9d3a-031331cb8482-istio-podinfo\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:32.275437 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.275262    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg4cm\" (UniqueName: \"kubernetes.io/projected/dcde29c5-e273-4dfa-9d3a-031331cb8482-kube-api-access-vg4cm\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:35:32.681509 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.681477    5875 generic.go:356] "Generic (PLEG): container finished" podID="dcde29c5-e273-4dfa-9d3a-031331cb8482" containerID="875c1052440e0df06673f6675e51e809c76ffea08ead739e6738d1f7d20552f9" exitCode=0
Nov 02 02:35:32.681695 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.681538    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"
Nov 02 02:35:32.681695 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.681560    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j" event={"ID":"dcde29c5-e273-4dfa-9d3a-031331cb8482","Type":"ContainerDied","Data":"875c1052440e0df06673f6675e51e809c76ffea08ead739e6738d1f7d20552f9"}
Nov 02 02:35:32.681695 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.681627    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j" event={"ID":"dcde29c5-e273-4dfa-9d3a-031331cb8482","Type":"ContainerDied","Data":"c88f99d8d9f47b686b0ce91e7f097d374c95e3d416f5669e971ffcf83ca5a7ec"}
Nov 02 02:35:32.681695 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.681651    5875 scope.go:122] "RemoveContainer" containerID="875c1052440e0df06673f6675e51e809c76ffea08ead739e6738d1f7d20552f9"
Nov 02 02:35:32.692586 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.692558    5875 scope.go:122] "RemoveContainer" containerID="28f0fc622e1dcd72511016a97c7381512fe9df2ff096abd5f635c4f1293395a0"
Nov 02 02:35:32.701032 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.701012    5875 scope.go:122] "RemoveContainer" containerID="875c1052440e0df06673f6675e51e809c76ffea08ead739e6738d1f7d20552f9"
Nov 02 02:35:32.701330 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:35:32.701313    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"875c1052440e0df06673f6675e51e809c76ffea08ead739e6738d1f7d20552f9\": container with ID starting with 875c1052440e0df06673f6675e51e809c76ffea08ead739e6738d1f7d20552f9 not found: ID does not exist" containerID="875c1052440e0df06673f6675e51e809c76ffea08ead739e6738d1f7d20552f9"
Nov 02 02:35:32.701395 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.701339    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"875c1052440e0df06673f6675e51e809c76ffea08ead739e6738d1f7d20552f9"} err="failed to get container status \"875c1052440e0df06673f6675e51e809c76ffea08ead739e6738d1f7d20552f9\": rpc error: code = NotFound desc = could not find container \"875c1052440e0df06673f6675e51e809c76ffea08ead739e6738d1f7d20552f9\": container with ID starting with 875c1052440e0df06673f6675e51e809c76ffea08ead739e6738d1f7d20552f9 not found: ID does not exist"
Nov 02 02:35:32.701395 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.701357    5875 scope.go:122] "RemoveContainer" containerID="28f0fc622e1dcd72511016a97c7381512fe9df2ff096abd5f635c4f1293395a0"
Nov 02 02:35:32.701602 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:35:32.701583    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f0fc622e1dcd72511016a97c7381512fe9df2ff096abd5f635c4f1293395a0\": container with ID starting with 28f0fc622e1dcd72511016a97c7381512fe9df2ff096abd5f635c4f1293395a0 not found: ID does not exist" containerID="28f0fc622e1dcd72511016a97c7381512fe9df2ff096abd5f635c4f1293395a0"
Nov 02 02:35:32.701661 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.701612    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f0fc622e1dcd72511016a97c7381512fe9df2ff096abd5f635c4f1293395a0"} err="failed to get container status \"28f0fc622e1dcd72511016a97c7381512fe9df2ff096abd5f635c4f1293395a0\": rpc error: code = NotFound desc = could not find container \"28f0fc622e1dcd72511016a97c7381512fe9df2ff096abd5f635c4f1293395a0\": container with ID starting with 28f0fc622e1dcd72511016a97c7381512fe9df2ff096abd5f635c4f1293395a0 not found: ID does not exist"
Nov 02 02:35:32.846819 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.846787    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"]
Nov 02 02:35:32.856528 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:32.856501    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-71e54d09-openshift-default-6987b8dff-45m8j"]
Nov 02 02:35:34.693184 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:34.693152    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcde29c5-e273-4dfa-9d3a-031331cb8482" path="/var/lib/kubelet/pods/dcde29c5-e273-4dfa-9d3a-031331cb8482/volumes"
Nov 02 02:35:36.560520 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:36.560484    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4"]
Nov 02 02:35:36.562029 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:36.561999    5875 cpu_manager.go:414] "containerMap: removing container" podUID="dcde29c5-e273-4dfa-9d3a-031331cb8482" containerName="istio-proxy"
Nov 02 02:35:36.562029 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:36.562030    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcde29c5-e273-4dfa-9d3a-031331cb8482" containerName="istio-proxy"
Nov 02 02:35:36.562305 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:36.562182    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcde29c5-e273-4dfa-9d3a-031331cb8482" containerName="istio-proxy"
Nov 02 02:35:36.562305 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:36.562204    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcde29c5-e273-4dfa-9d3a-031331cb8482" containerName="istio-proxy"
Nov 02 02:35:36.567183 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:36.567154    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4"
Nov 02 02:35:36.620275 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:36.620213    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrtl2\" (UniqueName: \"kubernetes.io/projected/077ff1e5-9289-4592-a3e8-9d96897b9f39-kube-api-access-wrtl2\") pod \"payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4\" (UID: \"077ff1e5-9289-4592-a3e8-9d96897b9f39\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4"
Nov 02 02:35:36.620425 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:36.620328    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/077ff1e5-9289-4592-a3e8-9d96897b9f39-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4\" (UID: \"077ff1e5-9289-4592-a3e8-9d96897b9f39\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4"
Nov 02 02:35:36.671260 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:36.671203    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-433aff\""
Nov 02 02:35:36.671260 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:36.671203    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-433aff-dockercfg-ch4lf\""
Nov 02 02:35:36.672948 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:36.672928    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4"]
Nov 02 02:35:36.721768 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:36.721734    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrtl2\" (UniqueName: \"kubernetes.io/projected/077ff1e5-9289-4592-a3e8-9d96897b9f39-kube-api-access-wrtl2\") pod \"payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4\" (UID: \"077ff1e5-9289-4592-a3e8-9d96897b9f39\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4"
Nov 02 02:35:36.721925 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:36.721775    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/077ff1e5-9289-4592-a3e8-9d96897b9f39-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4\" (UID: \"077ff1e5-9289-4592-a3e8-9d96897b9f39\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4"
Nov 02 02:35:36.722425 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:36.722406    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/077ff1e5-9289-4592-a3e8-9d96897b9f39-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4\" (UID: \"077ff1e5-9289-4592-a3e8-9d96897b9f39\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4"
Nov 02 02:35:36.783158 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:36.783121    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrtl2\" (UniqueName: \"kubernetes.io/projected/077ff1e5-9289-4592-a3e8-9d96897b9f39-kube-api-access-wrtl2\") pod \"payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4\" (UID: \"077ff1e5-9289-4592-a3e8-9d96897b9f39\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4"
Nov 02 02:35:36.885600 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:36.885524    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4"
Nov 02 02:35:37.264377 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:37.264356    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4"]
Nov 02 02:35:37.266680 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:35:37.266650    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod077ff1e5_9289_4592_a3e8_9d96897b9f39.slice/crio-d4fbbcd964e391ae389def576126dc96e28a8261e8712b7304a00f3771573ccc WatchSource:0}: Error finding container d4fbbcd964e391ae389def576126dc96e28a8261e8712b7304a00f3771573ccc: Status 404 returned error can't find the container with id d4fbbcd964e391ae389def576126dc96e28a8261e8712b7304a00f3771573ccc
Nov 02 02:35:37.708327 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:37.708293    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4" event={"ID":"077ff1e5-9289-4592-a3e8-9d96897b9f39","Type":"ContainerStarted","Data":"b5fb4c08d3c62f59d368f938e7a5684f7ce99769f4e27bde4548d5ada4da5fdf"}
Nov 02 02:35:37.708725 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:37.708350    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4" event={"ID":"077ff1e5-9289-4592-a3e8-9d96897b9f39","Type":"ContainerStarted","Data":"d4fbbcd964e391ae389def576126dc96e28a8261e8712b7304a00f3771573ccc"}
Nov 02 02:35:37.708725 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:37.708453    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4"
Nov 02 02:35:37.834459 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:37.834399    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4" podStartSLOduration=1.8343838940000001 podStartE2EDuration="1.834383894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:35:37.833741523 +0000 UTC m=+3785.274778021" watchObservedRunningTime="2026-11-02 02:35:37.834383894 +0000 UTC m=+3785.275420433"
Nov 02 02:35:41.318970 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:35:41.318932    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:35:42.837847 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:42.837808    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:35:48.715987 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:35:48.715956    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4"
Nov 02 02:36:11.317361 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:36:11.317316    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:36:37.100355 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:36:37.100320    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28798c31-qgz27"]
Nov 02 02:36:37.101672 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:36:37.101650    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28798c31-qgz27"]
Nov 02 02:36:38.693309 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:36:38.693276    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3" path="/var/lib/kubelet/pods/c3e835c0-7fb1-43ff-863a-ceef2e3ca8e3/volumes"
Nov 02 02:36:39.841784 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:36:39.841753    5875 scope.go:122] "RemoveContainer" containerID="d0e3cdfbb962e9b600ab5e01b459ab5eeff35b06b0c14a258f825c91d2207a66"
Nov 02 02:36:41.318493 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:36:41.318455    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:37:11.317353 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:37:11.317263    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:37:24.711605 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:37:24.711565    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4"]
Nov 02 02:37:29.712544 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:37:29.712481    5875 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4" podUID="077ff1e5-9289-4592-a3e8-9d96897b9f39" containerName="payload-processing" containerID="cri-o://b5fb4c08d3c62f59d368f938e7a5684f7ce99769f4e27bde4548d5ada4da5fdf" gracePeriod=25
Nov 02 02:37:29.967171 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:37:29.967108    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4"
Nov 02 02:37:30.095824 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:37:30.095787    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/077ff1e5-9289-4592-a3e8-9d96897b9f39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/077ff1e5-9289-4592-a3e8-9d96897b9f39-plugins-config-volume\") pod \"077ff1e5-9289-4592-a3e8-9d96897b9f39\" (UID: \"077ff1e5-9289-4592-a3e8-9d96897b9f39\") "
Nov 02 02:37:30.095989 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:37:30.095975    5875 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/077ff1e5-9289-4592-a3e8-9d96897b9f39-kube-api-access-wrtl2\" (UniqueName: \"kubernetes.io/projected/077ff1e5-9289-4592-a3e8-9d96897b9f39-kube-api-access-wrtl2\") pod \"077ff1e5-9289-4592-a3e8-9d96897b9f39\" (UID: \"077ff1e5-9289-4592-a3e8-9d96897b9f39\") "
Nov 02 02:37:30.096156 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:37:30.096135    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/077ff1e5-9289-4592-a3e8-9d96897b9f39-plugins-config-volume" pod "077ff1e5-9289-4592-a3e8-9d96897b9f39" (UID: "077ff1e5-9289-4592-a3e8-9d96897b9f39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:37:30.096345 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:37:30.096331    5875 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/077ff1e5-9289-4592-a3e8-9d96897b9f39-plugins-config-volume\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:37:30.098285 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:37:30.098253    5875 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/077ff1e5-9289-4592-a3e8-9d96897b9f39-kube-api-access-wrtl2" pod "077ff1e5-9289-4592-a3e8-9d96897b9f39" (UID: "077ff1e5-9289-4592-a3e8-9d96897b9f39"). InnerVolumeSpecName "kube-api-access-wrtl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:37:30.197669 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:37:30.197637    5875 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrtl2\" (UniqueName: \"kubernetes.io/projected/077ff1e5-9289-4592-a3e8-9d96897b9f39-kube-api-access-wrtl2\") on node \"ip-10-0-141-68.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:37:30.300512 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:37:30.300427    5875 generic.go:356] "Generic (PLEG): container finished" podID="077ff1e5-9289-4592-a3e8-9d96897b9f39" containerID="b5fb4c08d3c62f59d368f938e7a5684f7ce99769f4e27bde4548d5ada4da5fdf" exitCode=0
Nov 02 02:37:30.300512 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:37:30.300474    5875 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4"
Nov 02 02:37:30.300706 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:37:30.300515    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4" event={"ID":"077ff1e5-9289-4592-a3e8-9d96897b9f39","Type":"ContainerDied","Data":"b5fb4c08d3c62f59d368f938e7a5684f7ce99769f4e27bde4548d5ada4da5fdf"}
Nov 02 02:37:30.300706 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:37:30.300585    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4" event={"ID":"077ff1e5-9289-4592-a3e8-9d96897b9f39","Type":"ContainerDied","Data":"d4fbbcd964e391ae389def576126dc96e28a8261e8712b7304a00f3771573ccc"}
Nov 02 02:37:30.300706 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:37:30.300616    5875 scope.go:122] "RemoveContainer" containerID="b5fb4c08d3c62f59d368f938e7a5684f7ce99769f4e27bde4548d5ada4da5fdf"
Nov 02 02:37:30.310497 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:37:30.310475    5875 scope.go:122] "RemoveContainer" containerID="b5fb4c08d3c62f59d368f938e7a5684f7ce99769f4e27bde4548d5ada4da5fdf"
Nov 02 02:37:30.310806 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:37:30.310779    5875 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5fb4c08d3c62f59d368f938e7a5684f7ce99769f4e27bde4548d5ada4da5fdf\": container with ID starting with b5fb4c08d3c62f59d368f938e7a5684f7ce99769f4e27bde4548d5ada4da5fdf not found: ID does not exist" containerID="b5fb4c08d3c62f59d368f938e7a5684f7ce99769f4e27bde4548d5ada4da5fdf"
Nov 02 02:37:30.310931 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:37:30.310819    5875 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5fb4c08d3c62f59d368f938e7a5684f7ce99769f4e27bde4548d5ada4da5fdf"} err="failed to get container status \"b5fb4c08d3c62f59d368f938e7a5684f7ce99769f4e27bde4548d5ada4da5fdf\": rpc error: code = NotFound desc = could not find container \"b5fb4c08d3c62f59d368f938e7a5684f7ce99769f4e27bde4548d5ada4da5fdf\": container with ID starting with b5fb4c08d3c62f59d368f938e7a5684f7ce99769f4e27bde4548d5ada4da5fdf not found: ID does not exist"
Nov 02 02:37:30.488960 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:37:30.488926    5875 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4"]
Nov 02 02:37:30.538506 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:37:30.538479    5875 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-433aff-569dd7865-9qnd4"]
Nov 02 02:37:30.693489 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:37:30.693415    5875 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="077ff1e5-9289-4592-a3e8-9d96897b9f39" path="/var/lib/kubelet/pods/077ff1e5-9289-4592-a3e8-9d96897b9f39/volumes"
Nov 02 02:37:39.861655 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:37:39.861623    5875 scope.go:122] "RemoveContainer" containerID="cac45eff0393c0500e3fcd9d846a3005ae71f9c752762af9f3f8fbacbe5bb8d9"
Nov 02 02:37:40.384569 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:37:40.384536    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:37:41.318757 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:37:41.318721    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:37:54.511175 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:37:54.511142    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:38:07.597901 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:38:07.597870    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:38:11.317532 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:38:11.317491    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:38:28.664691 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:38:28.664653    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:38:39.886355 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:38:39.886318    5875 scope.go:122] "RemoveContainer" containerID="06cf501488ab21bab39307a543f0b4318202e250576ee642571c2688f7d37e1a"
Nov 02 02:38:41.317497 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:38:41.317456    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:39:01.180892 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:01.180857    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xgf4f"]
Nov 02 02:39:11.319868 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:39:11.319830    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:39:11.456872 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:11.456845    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7bbc5d984c-7w9xv_9810c039-61bd-483f-9b96-857fbcb2f633/manager/0.log"
Nov 02 02:39:13.055369 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:13.055338    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f8c46b7cf-5584m_6ba1716a-8773-45a5-902e-9098578180bf/manager/0.log"
Nov 02 02:39:15.240488 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:15.240461    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6d4c89cd86-fcs9s_a7381a4a-d748-49f1-993c-cb34b0254e49/maas-api/0.log"
Nov 02 02:39:19.202661 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:19.202627    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-sfvtj_8c9b0d42-b9de-4e90-a03b-dd3b73e8e209/kuadrant-console-plugin/0.log"
Nov 02 02:39:19.816794 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:19.816761    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-xgf4f_2bfb1fba-9189-4c25-9805-ebe4bc75da01/limitador/0.log"
Nov 02 02:39:21.110925 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:21.110884    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-t8vnh_4ed38132-83a1-4cc5-9545-c1623a3b003f/discovery/0.log"
Nov 02 02:39:21.471956 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:21.471916    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-bf99ddbdb-jkkrf_8e0923dd-1994-424e-93c7-dd9cd1431dc5/kube-auth-proxy/0.log"
Nov 02 02:39:22.245862 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:22.245829    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-727fm_a50af6e0-a390-47d4-a4d2-7f90a3055661/istio-proxy/0.log"
Nov 02 02:39:23.398311 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:23.398278    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-ghslr_0c2d802d-8fbd-430d-ae72-29b0e217714d/istio-proxy/0.log"
Nov 02 02:39:24.030518 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:24.030487    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-h8px6_ba032348-8064-40c7-8278-0115c943882e/istio-proxy/0.log"
Nov 02 02:39:24.345075 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:24.344995    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-mfmmj_004e401a-4d50-46ec-b1bd-d9896f9b20d1/istio-proxy/0.log"
Nov 02 02:39:26.738141 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:26.738111    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-fxwst_2d0bed6b-fa05-48dd-8f55-21d4903222c9/main/0.log"
Nov 02 02:39:27.293425 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:27.293384    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-k4d5r_368af2aa-46fd-4d17-b1c7-fcb9122e1632/main/0.log"
Nov 02 02:39:27.583971 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:27.583889    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn_61e08deb-d43d-4418-8c8d-414c18fa0916/main/0.log"
Nov 02 02:39:38.467132 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:38.467099    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7bbc5d984c-7w9xv_9810c039-61bd-483f-9b96-857fbcb2f633/manager/0.log"
Nov 02 02:39:39.835661 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:39.835629    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f8c46b7cf-5584m_6ba1716a-8773-45a5-902e-9098578180bf/manager/0.log"
Nov 02 02:39:41.317820 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:39:41.317778    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:39:41.501039 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:41.501002    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6d4c89cd86-fcs9s_a7381a4a-d748-49f1-993c-cb34b0254e49/maas-api/0.log"
Nov 02 02:39:44.870306 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:44.870275    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-sfvtj_8c9b0d42-b9de-4e90-a03b-dd3b73e8e209/kuadrant-console-plugin/0.log"
Nov 02 02:39:45.349077 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:45.349044    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-xgf4f_2bfb1fba-9189-4c25-9805-ebe4bc75da01/limitador/0.log"
Nov 02 02:39:46.414634 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:46.414591    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-t8vnh_4ed38132-83a1-4cc5-9545-c1623a3b003f/discovery/0.log"
Nov 02 02:39:46.760582 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:46.760553    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-bf99ddbdb-jkkrf_8e0923dd-1994-424e-93c7-dd9cd1431dc5/kube-auth-proxy/0.log"
Nov 02 02:39:47.495185 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:47.495149    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-727fm_a50af6e0-a390-47d4-a4d2-7f90a3055661/istio-proxy/0.log"
Nov 02 02:39:48.652433 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:48.652385    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-ghslr_0c2d802d-8fbd-430d-ae72-29b0e217714d/istio-proxy/0.log"
Nov 02 02:39:49.292030 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:49.291999    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-h8px6_ba032348-8064-40c7-8278-0115c943882e/istio-proxy/0.log"
Nov 02 02:39:49.621446 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:49.621360    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-mfmmj_004e401a-4d50-46ec-b1bd-d9896f9b20d1/istio-proxy/0.log"
Nov 02 02:39:51.811822 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:51.811791    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-fxwst_2d0bed6b-fa05-48dd-8f55-21d4903222c9/main/0.log"
Nov 02 02:39:52.340059 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:52.340032    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-k4d5r_368af2aa-46fd-4d17-b1c7-fcb9122e1632/main/0.log"
Nov 02 02:39:52.594685 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:39:52.594595    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b849bbn_61e08deb-d43d-4418-8c8d-414c18fa0916/main/0.log"
Nov 02 02:40:11.317627 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:40:11.317539    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:40:21.341323 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:40:21.341289    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f8c46b7cf-5584m_6ba1716a-8773-45a5-902e-9098578180bf/manager/0.log"
Nov 02 02:40:21.908432 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:40:21.908397    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f8c46b7cf-5584m_6ba1716a-8773-45a5-902e-9098578180bf/manager/0.log"
Nov 02 02:40:41.317874 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:40:41.317837    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:40:58.452963 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:40:58.452930    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f8c46b7cf-5584m_6ba1716a-8773-45a5-902e-9098578180bf/manager/0.log"
Nov 02 02:40:59.036792 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:40:59.036756    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f8c46b7cf-5584m_6ba1716a-8773-45a5-902e-9098578180bf/manager/0.log"
Nov 02 02:41:09.139149 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:09.139113    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8vhjc_b88b4dfb-1523-4701-9388-781cedab823b/global-pull-secret-syncer/0.log"
Nov 02 02:41:11.318784 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:41:11.318742    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:41:12.727532 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:12.727499    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ft5bg_65f63e43-859a-4d3a-8393-99724782596b/konnectivity-agent/0.log"
Nov 02 02:41:14.564862 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:14.564833    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-68.us-west-1.compute.internal_5845b2d4146c451ff8e611722d7c3e34/haproxy/0.log"
Nov 02 02:41:19.132977 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.132936    5875 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cfrs6/perf-node-gather-daemonset-982f6"]
Nov 02 02:41:19.133469 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.133454    5875 cpu_manager.go:414] "containerMap: removing container" podUID="077ff1e5-9289-4592-a3e8-9d96897b9f39" containerName="payload-processing"
Nov 02 02:41:19.133523 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.133471    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="077ff1e5-9289-4592-a3e8-9d96897b9f39" containerName="payload-processing"
Nov 02 02:41:19.133567 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.133521    5875 cpu_manager.go:414] "containerMap: removing container" podUID="dcde29c5-e273-4dfa-9d3a-031331cb8482" containerName="istio-proxy"
Nov 02 02:41:19.133567 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.133528    5875 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcde29c5-e273-4dfa-9d3a-031331cb8482" containerName="istio-proxy"
Nov 02 02:41:19.133647 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.133604    5875 memory_manager.go:365] "RemoveStaleState removing state" podUID="077ff1e5-9289-4592-a3e8-9d96897b9f39" containerName="payload-processing"
Nov 02 02:41:19.139205 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.139182    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-982f6"
Nov 02 02:41:19.191121 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.191087    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cfrs6\"/\"kube-root-ca.crt\""
Nov 02 02:41:19.191326 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.191145    5875 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cfrs6\"/\"openshift-service-ca.crt\""
Nov 02 02:41:19.201384 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.201354    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef61016c-6f52-424a-8b08-0706d1563cc0-lib-modules\") pod \"perf-node-gather-daemonset-982f6\" (UID: \"ef61016c-6f52-424a-8b08-0706d1563cc0\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-982f6"
Nov 02 02:41:19.201562 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.201410    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef61016c-6f52-424a-8b08-0706d1563cc0-sys\") pod \"perf-node-gather-daemonset-982f6\" (UID: \"ef61016c-6f52-424a-8b08-0706d1563cc0\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-982f6"
Nov 02 02:41:19.201562 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.201535    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ef61016c-6f52-424a-8b08-0706d1563cc0-podres\") pod \"perf-node-gather-daemonset-982f6\" (UID: \"ef61016c-6f52-424a-8b08-0706d1563cc0\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-982f6"
Nov 02 02:41:19.201667 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.201627    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ef61016c-6f52-424a-8b08-0706d1563cc0-proc\") pod \"perf-node-gather-daemonset-982f6\" (UID: \"ef61016c-6f52-424a-8b08-0706d1563cc0\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-982f6"
Nov 02 02:41:19.201716 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.201665    5875 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxbpp\" (UniqueName: \"kubernetes.io/projected/ef61016c-6f52-424a-8b08-0706d1563cc0-kube-api-access-qxbpp\") pod \"perf-node-gather-daemonset-982f6\" (UID: \"ef61016c-6f52-424a-8b08-0706d1563cc0\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-982f6"
Nov 02 02:41:19.240853 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.240814    5875 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cfrs6\"/\"default-dockercfg-ppkg9\""
Nov 02 02:41:19.242750 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.242668    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cfrs6/perf-node-gather-daemonset-982f6"]
Nov 02 02:41:19.302548 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.302509    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef61016c-6f52-424a-8b08-0706d1563cc0-sys\") pod \"perf-node-gather-daemonset-982f6\" (UID: \"ef61016c-6f52-424a-8b08-0706d1563cc0\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-982f6"
Nov 02 02:41:19.302801 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.302605    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ef61016c-6f52-424a-8b08-0706d1563cc0-podres\") pod \"perf-node-gather-daemonset-982f6\" (UID: \"ef61016c-6f52-424a-8b08-0706d1563cc0\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-982f6"
Nov 02 02:41:19.302801 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.302652    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef61016c-6f52-424a-8b08-0706d1563cc0-sys\") pod \"perf-node-gather-daemonset-982f6\" (UID: \"ef61016c-6f52-424a-8b08-0706d1563cc0\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-982f6"
Nov 02 02:41:19.302801 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.302687    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ef61016c-6f52-424a-8b08-0706d1563cc0-proc\") pod \"perf-node-gather-daemonset-982f6\" (UID: \"ef61016c-6f52-424a-8b08-0706d1563cc0\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-982f6"
Nov 02 02:41:19.302801 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.302738    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ef61016c-6f52-424a-8b08-0706d1563cc0-podres\") pod \"perf-node-gather-daemonset-982f6\" (UID: \"ef61016c-6f52-424a-8b08-0706d1563cc0\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-982f6"
Nov 02 02:41:19.302801 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.302744    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ef61016c-6f52-424a-8b08-0706d1563cc0-proc\") pod \"perf-node-gather-daemonset-982f6\" (UID: \"ef61016c-6f52-424a-8b08-0706d1563cc0\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-982f6"
Nov 02 02:41:19.302801 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.302797    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxbpp\" (UniqueName: \"kubernetes.io/projected/ef61016c-6f52-424a-8b08-0706d1563cc0-kube-api-access-qxbpp\") pod \"perf-node-gather-daemonset-982f6\" (UID: \"ef61016c-6f52-424a-8b08-0706d1563cc0\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-982f6"
Nov 02 02:41:19.303146 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.302940    5875 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef61016c-6f52-424a-8b08-0706d1563cc0-lib-modules\") pod \"perf-node-gather-daemonset-982f6\" (UID: \"ef61016c-6f52-424a-8b08-0706d1563cc0\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-982f6"
Nov 02 02:41:19.303146 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.303043    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef61016c-6f52-424a-8b08-0706d1563cc0-lib-modules\") pod \"perf-node-gather-daemonset-982f6\" (UID: \"ef61016c-6f52-424a-8b08-0706d1563cc0\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-982f6"
Nov 02 02:41:19.363864 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.363827    5875 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxbpp\" (UniqueName: \"kubernetes.io/projected/ef61016c-6f52-424a-8b08-0706d1563cc0-kube-api-access-qxbpp\") pod \"perf-node-gather-daemonset-982f6\" (UID: \"ef61016c-6f52-424a-8b08-0706d1563cc0\") " pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-982f6"
Nov 02 02:41:19.451879 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.451798    5875 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-982f6"
Nov 02 02:41:19.823558 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.823534    5875 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cfrs6/perf-node-gather-daemonset-982f6"]
Nov 02 02:41:19.826416 ip-10-0-141-68 kubenswrapper[5875]: W1102 02:41:19.826386    5875 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podef61016c_6f52_424a_8b08_0706d1563cc0.slice/crio-3a8d359b7e30a3274813fd7aca3f2bddfa2f8e63b2a2ab22eceedebc1d0a9a05 WatchSource:0}: Error finding container 3a8d359b7e30a3274813fd7aca3f2bddfa2f8e63b2a2ab22eceedebc1d0a9a05: Status 404 returned error can't find the container with id 3a8d359b7e30a3274813fd7aca3f2bddfa2f8e63b2a2ab22eceedebc1d0a9a05
Nov 02 02:41:19.837617 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:19.837597    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-sfvtj_8c9b0d42-b9de-4e90-a03b-dd3b73e8e209/kuadrant-console-plugin/0.log"
Nov 02 02:41:20.552796 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:20.552756    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-982f6" event={"ID":"ef61016c-6f52-424a-8b08-0706d1563cc0","Type":"ContainerStarted","Data":"ffa250c08d54ea55af749d1f582a9c23b4db3f5c821ef15a7ffca91a55a30cc1"}
Nov 02 02:41:20.553216 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:20.552836    5875 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-982f6" event={"ID":"ef61016c-6f52-424a-8b08-0706d1563cc0","Type":"ContainerStarted","Data":"3a8d359b7e30a3274813fd7aca3f2bddfa2f8e63b2a2ab22eceedebc1d0a9a05"}
Nov 02 02:41:20.553216 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:20.552957    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-982f6"
Nov 02 02:41:20.668210 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:20.667997    5875 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-982f6" podStartSLOduration=1.667984353 podStartE2EDuration="1.667984353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:41:20.667496157 +0000 UTC m=+4128.108532657" watchObservedRunningTime="2026-11-02 02:41:20.667984353 +0000 UTC m=+4128.109020896"
Nov 02 02:41:20.679839 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:20.679813    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-xgf4f_2bfb1fba-9189-4c25-9805-ebe4bc75da01/limitador/0.log"
Nov 02 02:41:23.690030 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:23.689993    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b1d7626a-fd60-4593-8a06-8740ed75893f/alertmanager/0.log"
Nov 02 02:41:24.105880 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:24.105851    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b1d7626a-fd60-4593-8a06-8740ed75893f/config-reloader/0.log"
Nov 02 02:41:24.519216 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:24.519185    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b1d7626a-fd60-4593-8a06-8740ed75893f/kube-rbac-proxy-web/0.log"
Nov 02 02:41:24.934147 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:24.934052    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b1d7626a-fd60-4593-8a06-8740ed75893f/kube-rbac-proxy/0.log"
Nov 02 02:41:25.350747 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:25.350722    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b1d7626a-fd60-4593-8a06-8740ed75893f/kube-rbac-proxy-metric/0.log"
Nov 02 02:41:25.765234 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:25.765204    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b1d7626a-fd60-4593-8a06-8740ed75893f/prom-label-proxy/0.log"
Nov 02 02:41:26.184550 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:26.184453    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b1d7626a-fd60-4593-8a06-8740ed75893f/init-config-reloader/0.log"
Nov 02 02:41:26.569248 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:26.569196    5875 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cfrs6/perf-node-gather-daemonset-982f6"
Nov 02 02:41:27.131978 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:27.131942    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-vhh4r_5f70370d-3793-4e25-90e8-fc252f5a7596/kube-state-metrics/0.log"
Nov 02 02:41:27.539622 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:27.539587    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-vhh4r_5f70370d-3793-4e25-90e8-fc252f5a7596/kube-rbac-proxy-main/0.log"
Nov 02 02:41:27.954865 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:27.954839    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-vhh4r_5f70370d-3793-4e25-90e8-fc252f5a7596/kube-rbac-proxy-self/0.log"
Nov 02 02:41:28.374377 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:28.374305    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-79ffdc6779-dp2ff_544ee005-9680-4d16-ae14-a2633763bf10/metrics-server/0.log"
Nov 02 02:41:28.790774 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:28.790742    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-rc9qm_e9dd01bc-7209-4fe5-ac79-8efaa6c14232/monitoring-plugin/0.log"
Nov 02 02:41:30.470334 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:30.470309    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rrtpw_9bbd7104-ba74-4c31-8cfb-3f307da4b3f4/node-exporter/0.log"
Nov 02 02:41:30.882375 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:30.882294    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rrtpw_9bbd7104-ba74-4c31-8cfb-3f307da4b3f4/kube-rbac-proxy/0.log"
Nov 02 02:41:31.299139 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:31.299109    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rrtpw_9bbd7104-ba74-4c31-8cfb-3f307da4b3f4/init-textfile/0.log"
Nov 02 02:41:32.972250 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:32.972204    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-4snk7_f86263ab-30e1-40bf-a671-f82f44423125/kube-rbac-proxy-main/0.log"
Nov 02 02:41:33.389604 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:33.389529    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-4snk7_f86263ab-30e1-40bf-a671-f82f44423125/kube-rbac-proxy-self/0.log"
Nov 02 02:41:33.805011 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:33.804983    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-4snk7_f86263ab-30e1-40bf-a671-f82f44423125/openshift-state-metrics/0.log"
Nov 02 02:41:34.229568 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:34.229544    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6625c4e9-083d-4c39-9bb8-e3398cc9f68b/prometheus/0.log"
Nov 02 02:41:34.648399 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:34.648317    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6625c4e9-083d-4c39-9bb8-e3398cc9f68b/config-reloader/0.log"
Nov 02 02:41:35.066794 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:35.066764    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6625c4e9-083d-4c39-9bb8-e3398cc9f68b/thanos-sidecar/0.log"
Nov 02 02:41:35.482019 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:35.481987    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6625c4e9-083d-4c39-9bb8-e3398cc9f68b/kube-rbac-proxy-web/0.log"
Nov 02 02:41:35.896095 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:35.896018    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6625c4e9-083d-4c39-9bb8-e3398cc9f68b/kube-rbac-proxy/0.log"
Nov 02 02:41:36.311535 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:36.311502    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6625c4e9-083d-4c39-9bb8-e3398cc9f68b/kube-rbac-proxy-thanos/0.log"
Nov 02 02:41:36.726589 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:36.726562    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6625c4e9-083d-4c39-9bb8-e3398cc9f68b/init-config-reloader/0.log"
Nov 02 02:41:37.149575 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:37.149475    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-8zqhk_792b1392-5f44-47a8-9b86-05ee811046ea/prometheus-operator/0.log"
Nov 02 02:41:37.556771 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:37.556739    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-8zqhk_792b1392-5f44-47a8-9b86-05ee811046ea/kube-rbac-proxy/0.log"
Nov 02 02:41:37.974991 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:37.974956    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-kpwsv_85999952-cf96-48d1-a608-91c84a7bc17c/prometheus-operator-admission-webhook/0.log"
Nov 02 02:41:38.396079 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:38.396012    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b76c759df-p6cl8_9df8588e-97f8-4931-ad82-a5b4b2b02407/telemeter-client/0.log"
Nov 02 02:41:38.813206 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:38.813171    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b76c759df-p6cl8_9df8588e-97f8-4931-ad82-a5b4b2b02407/reload/0.log"
Nov 02 02:41:39.229121 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:39.229091    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b76c759df-p6cl8_9df8588e-97f8-4931-ad82-a5b4b2b02407/kube-rbac-proxy/0.log"
Nov 02 02:41:39.653081 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:39.653006    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b84687bcd-x24hf_c333281d-bd01-4c24-bb15-fa3ee2583fb9/thanos-query/0.log"
Nov 02 02:41:40.064835 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:40.064761    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b84687bcd-x24hf_c333281d-bd01-4c24-bb15-fa3ee2583fb9/kube-rbac-proxy-web/0.log"
Nov 02 02:41:40.488688 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:40.488664    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b84687bcd-x24hf_c333281d-bd01-4c24-bb15-fa3ee2583fb9/kube-rbac-proxy/0.log"
Nov 02 02:41:40.902839 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:40.902760    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b84687bcd-x24hf_c333281d-bd01-4c24-bb15-fa3ee2583fb9/prom-label-proxy/0.log"
Nov 02 02:41:41.317954 ip-10-0-141-68 kubenswrapper[5875]: E1102 02:41:41.317916    5875 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4njj" containerName="extractor"
Nov 02 02:41:41.318514 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:41.318492    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b84687bcd-x24hf_c333281d-bd01-4c24-bb15-fa3ee2583fb9/kube-rbac-proxy-rules/0.log"
Nov 02 02:41:41.733080 ip-10-0-141-68 kubenswrapper[5875]: I1102 02:41:41.733050    5875 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b84687bcd-x24hf_c333281d-bd01-4c24-bb15-fa3ee2583fb9/kube-rbac-proxy-metrics/0.log"